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1.  Introduction 


As  the  Department  of  Defense  (DOD)  moves  toward  addressing  geopolitieal  envi¬ 
ronments  marked  by  rapidly  changing  threats,  tactics,  mission  scenarios,  technolo¬ 
gies,  and  available  funding,  advanced  methods,  processes,  and  tools  are  needed  to 
effectively  engineer  resilient  system  solutions.  The  resulting  resilient  systems  must 
be  adaptable  to  a  wider  range  of  mission  contexts,  across  multiple  alternative  fu¬ 
tures.  To  support  resilient  system  design,  and  a  correspondingly  resilient  design  pro¬ 
cess,  tradespace  exploration  (TSE)  provides  decision  makers  an  understanding  of 
capabilities,  gaps,  and  potential  compromises  facilitating  the  achievement  of  system 
metric  objectives.  The  decisions  being  made  throughout  a  system’s  development 
life  cycle  are  continuously  redefining  its  capabilities,  performance,  cost,  manufac¬ 
turability,  delivery  schedule,  and  sustainability.  To  be  effective,  decision  makers 
must  have  deep  knowledge  of  the  component  elements  of  a  system,  including  how 
these  elements  interact  internally  to  the  system  and  externally  with  the  operational 
environment.  TSE  provides  decision  makers  an  understanding  of  candidate  system 
component  choices  and  the  implications  on  multiple  missions  across  joint  Warfight¬ 
ing  environments.' 

Throughout  a  system’s  development  life  cycle,  and  across  its  hierarchy,  trades  are 
being  performed  with  multiple  types  and  quantities  of  data  coming  out  of  varying 
levels  of  analysis  fidelity.  Erom  a  system’s  functional  perspective,  capabilities  (e.g., 
lighter  weight,  higher  performing)  map  to  system  measures  of  performance,  which 
in  turn  map  to  measures  of  effectiveness,  which  ultimately  affect  operational  fig¬ 
ures  of  merit  that  determine  how  well  a  system,  or  a  portfolio  of  systems,  meets 
a  required  capability.  Concurrently,  there  is  a  product  hierarchy  within  which  var¬ 
ious  technologies  can  be  applied,  from  the  component  level,  and  up  to  subsystem, 
system,  and  finally  a  portfolio  of  heterogeneous  systems  working  concurrently  to 
fulfill  a  role.  Additionally,  there  are  process  considerations  such  as  industrial  base, 
training,  policy,  and  procedures,  which  are  often  omitted  from  the  decision  analysis 
process  for  various  reasons.  Although  the  decisions  being  made  throughout  these 
functional,  product,  and  process  perspectives  differ,  decision  makers  at  the  highest 
levels  will  execute  based  on  the  relationship  between  the  benefit  achieved  by,  and 
the  life  cycle  cost  (ECC)  associated  with,  a  particular  asset  mix.  Therefore,  if  the 
tradeoff  between  benefit  and  cost  can  be  generated,  explored,  and  then  presented  in 
the  context  of  the  holistic  system,  a  more  informed  decision  can  be  made  across  the 
system  perspectives. 
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To  aid  the  decision  maker  in  performing  effective  TSE  under  these  complex  circum¬ 
stances,  the  Engineered  Resilient  Systems  (ERS)  effort  involves  providing  the  nec¬ 
essary  engineering  concepts,  methods,  processes,  and  tools. ^  The  goals  of  ERS  are 
to  provide  to  engineering.  Warfighting,  and  acquisition  decision  makers  the  needed 
capability  to  manage  TSE  activities  with  full  and  consistent  information  throughout 
the  life  of  the  systems  by  the  following^: 

•  Producing  more  complete  and  robust  requirements  pre-Milestone  A 

•  Making  the  engineering  design  process  much  more  efficient  and  effective 

•  Considering  the  manufacturability  of  a  proposed  design  explicitly 

•  Establishing  baseline  resiliency  of  current  capabilities 

The  TSE  methods,  processes,  and  tools  should  enable  deeper  consideration  of  sys¬ 
tem  design  alternatives  while  keeping  the  space  as  open  as  possible  to  address  re¬ 
siliency  and  robustness  to  changing  conditions  and  constraints. 

The  tradespace  frontier  of  Eig.  C  depicts  that  information  coming  out  of  knowledge 
databases  must  support  decision  making  across  the  life  cycle  by  communicating  to 
multiple  perspectives  and  across  the  system  hierarchy,  while  taking  into  account  fis¬ 
cal  and  environmental  constraints.  What  this  means  is  that  TSE  tools  and  processes 
are  needed  at  different  levels  of  the  product  hierarchy,  commensurate  with  the  de¬ 
cisions  being  made,  the  information  available  to  make  decisions,  and  the  person  or 
organization  making  decisions. 
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Fig.  1  ERS  tradespace  frontier 
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2.  Assumptions 


•  TSE  is  performed  throughout  a  system’s  life  eyele,  in  support  of  deeision 
making,  in  a  step-by-step  proeess. 

•  There  exist  organization-speeifie  best  praetiee  methods,  proeesses,  and  tools 
for  performing  TSE. 

•  TSE  is  often  performed  to  justify  seleetion  of  a  redueed  set  of  options  from 
a  broader  set  of  alternatives,  rather  than  for  understanding  how  eompromises 
in  system  design  ean  inerease  value  delivered  to  stakeholders. 

•  TSE  is  largely  performed  ad  hoe,  and  with  the  tools  on-hand,  without  inves¬ 
tigation  into  how  other  tools  ean  be  brought  in  to  fill  gaps  in  a  TSE  proeess. 

•  TSE  is  performed  to  "make  better  deeisions",  but  without  eriteria  for  mea¬ 
suring  the  effeetiveness,  effieieney,  eorreetness,  or  eonsisteney  of  deeisions 
aeross  the  life  eyele. 

3.  Approach  

The  approaeh  to  reveal  whether  TSE  best  praetiees  are  on  a  trajeetory  to  meet  the 
needs  of  ERS  involves  the  following:  1)  sourees  of  TSE  methods,  proeesses,  and 
tools  were  identified,  2)  eommon  approaehes  in  TSE  were  doeumented,  3)  a  set 
of  TSE  tool  requirements  was  developed,  4)  the  most  eommonly  used  TSE  tools, 
as  well  as  other  tools  that  eould  be  implemented  in  a  formal  TSE  proeess,  were 
identified,  and  5)  an  evaluation  was  performed. 

When  making  a  deeision,  in  this  ease  regarding  the  tools  to  use  in  exploring  a 
tradespaee,  a  eommon  approaeh  is  to  start  with  alternatives  and  determine  whieh  al¬ 
ternative  best  meets  the  deeision  maker’s  intent.  A  preferred  approaeh  is  one  based 
on  value-foeused  thinking  (VET),  where  value-produeing  objeetives  are  developed, 
alternatives  are  identified,  and  deeisions  are  made  based  on  how  mueh  value  the 
alternatives  provide  aeross  the  objeetives.^  Applied  formally,  a  VET  approaeh  in¬ 
volves  interviews  with  deeision  makers  to  help  translate  the  pereeived  inerease  in 
an  evaluation  metrie  into  a  value  eurve.  In  this  study,  a  modified  VET  approaeh 
was  started;  a  reeommendation  will  be  made  for  expanding  the  eurrent  approaeh  to 
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include  generating  value  eurves  and  assigning  swing  weights  to  value  measures  for 
applieation  to  an  additive  value  model. 

To  initiate  a  VFT  approaeh,  a  fundamental  objeetive  was  developed  assuming  that 
TSE  is  a  proeess  that  uses  tools  to  arrive  at  deeisions: 

Use  a  tradespace  tool,  or  set  of  tools,  to  make  actionable  decisions  across  the 
system  hierarchy  and  throughout  the  system  life  cycle. 

To  evaluate  a  tool’s  performanee  within  a  TSE  proeess,  several  pieees  of  informa¬ 
tion  were  needed:  TSE  tools,  TSE  proeess  steps,  attributes  that  define  TSE  tools, 
and  attributes  that  define  what  users  are  doing  with  their  TSE  tools. 

4.  Tradespace  Analysis  Tool  Requirements  Identification 

An  audit  was  performed  to  identify  the  existing  TSE  tools  used  by  the  ERS  Demon¬ 
stration  Projeets  (DPs),^  as  well  as  elsewhere  in  the  serviees  and  industry,  through¬ 
out  a  system’s  life  eyele.  A  set  of  attributes  that  define  various  TSE  tools  was  de¬ 
veloped,  and  the  tools  were  mapped  to  the  attributes  to  aid  in  identifying  existing 
tools  that  are  elosely  aligned  with  needed  eapabilities. 

4.1  TSE  Tools 

The  following  sourees  were  used  to  identify  TSE  tools: 

•  Operations  Researeh/Management  Seienees  (OR/MS)  Today  Deeision  Analysis 
Software  (DAS)  Survey^’^ 

•  Rexer  Analyties  Data  Miner  Survey’ 

•  ERS  DPs^ 

•  ERS  Priority  Steering  Couneil  (PSC)  Tools  Assessments^ 

•  US  Army  Researeh  Eaboratory  (ARE)  internal  organization  researeh  and  ex- 
perienee 

After  eompleting  the  audit  of  tools,  it  was  apparent  that  what  most  refer  to  as  a 
deeision  analysis  or  data  mining  tool  eould  be  miseonstrued  as  a  satisfaetory  ERS 
TSE  tool.  To  better  understand  how  a  deeision  analysis  tool  is  viewed,  a  definition 
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of  decision  analysis  developed  by  Buede/^  and  referred  to  in  the  2010  OR/MS 
Today  survey  summary,^  is  provided: 

"Decision  analysis  is  the  discipline  of  evaluating  complex  alternatives  in  the  light 
of  uncertainty,  value  preferences  and  risk  preference." 

Although  the  process  of  decision  analysis,  as  described  in  the  Defense  Acquisition 
University’s  Defense  Acquisition  Guidebook,^^  is  not  TSE  in  and  of  itself,  it  is 
supported  by  TSE.  In  the  case  of  data  mining  tools,  the  corresponding  Data  Miner 
Survey’  does  not  provide  a  formal  definition,  but  similarities  to  TSE  activities  are 
apparent  with  terms  such  as  regression,  models,  and  data  visualization. 

The  ERS  PSC  assessments^  was  filtered  for  tools  in  the  "Data  Driven  Tradespace 
Exploration  and  Analysis"  section.  In  addition  to  the  source  materials  in  Section 
4.1,  the  ERS  DP  leads  provided  overview  presentations  and  were  informally  inter¬ 
viewed  via  telephone.  These  sources  all  revealed  that  there  are  many  tools  in  use 
across  academia,  government,  and  industry  for  performing  TSE.  In  all,  81  tools 
were  identified,  shown  in  Table  1  (the  authors  recognize  that  more  tools  exist  than 
are  captured  here).  A  short  description  of  each  tool  is  provided  in  Appendix  A. 

Table  1  TSE  tools  and  developers 


Tool/Process 

Developer 

Tool/Process 

Developer 

©RISK 

Palisade  Corporation 

Mathematica 

Wolfram 

AAMODAT 

ARDEC 

MedModel  Suite 

ProModel  Corporation 

AgenaRisk 

Agena  Limited 

modeERONTIER 

Esteco 

Analytica 

Lumina  Decision 

Systems 

ModelCenter 

Phoenix  Integration 

Analytics 

SAS 

Nexus 

iChrome  Ltd. 

ASEC 

TARDEC 

OpenMDAO 

NASA 

ATSV 

PSU-ARL 

OptiY 

OptiY  GmbH 

Berkeley 

University  of  California 

Portfolio 

ProModel  Corporation 

Madonna 

at  Berkeley 

Simulator 

CART 

Salford  Systems 

Process  Simulator 

ProModel  Corporation 

ClearPoint 

Ascendant  Strategy 

Project  Simulator 

ProModel  Corporation 

Strategy 

Management  Group 

Comparion  Suite 

Expert  Choice,  Inc. 

Promax 

Cogentus  Consulting 

Professional 

Ltd 

CPAT 

PEO-GCS 

ProModel  Suite 

ProModel  Corporation 

CyDesign  Studio 

CyDesign  Labs 

R 

R  Eoundation 

D3 

D3JS.org 

RandomEorests 

Salford  Systems 

DAKOTA 

Sandia  National  Labs 

Rave 

Georgia  Tech 

DEA  SolverPro 

SAITECH,  Inc. 

RICH  Decisions 

SAL,  Aalto  University 

DecidelT 

Preference 

Risk  Solver  Pro 

Erontline  Systems,  Inc. 

Decision  Explorer 

Banxia  Software  Ltd 

RiskSim 

TreePlan  Software 
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Table  1  TSE  tools  and  developers  (continued) 


Tool/Process 

Developer 

Tool/Process 

Developer 

DecisionTools 

Suite  6.0 

Palisade  Corporation 

ROSETTA 

ASDL,  Georgia  Tech 

D-Sight  Desktop 

D-Sight 

RPM  Decisions 

SAL,  Aalto  University 

Enterprise 

Portfolio 

Simulator 

ProModel  Corporation 

SAE 

MITRE 

Equity3 

Catalyze  Ltd 

Salford  Predictive 

Modeler 

Salford  Systems 

Eureqa  Desktop 

Nutonian,  Inc. 

SCAP 

SLAD 

Excel 

Microsoft 

SensIt 

TreePlan  Software 

ExtendSim  Suite 

Imagine  That,  Inc. 

ServiceModel 

ProModel  Corporation 

EACT 

GTRI 

SIMULIA 

Dassault  Systemes 

EoreTell-DSS 

DecisionPath,  Inc. 

Smart-Swaps 

SAL,  Aalto  University 

GoldSim 

GoldSim  Technology 
Group 

SPIDR 

University  of  Southern 
California 

gPROMS 

Process  Systems 
Enterprise  Limited 

STATISTICA 

StatSoft 

GRIPS 

The  Aerospace  Corp 

Tableau 

tableau  software 

Hiview3 

Catalyze  Ltd 

TIES 

ASDL,  Georgia  Tech 

Iris 

Ayasdi 

TRACER 

LMI 

Isight 

Dassault  Systemes 

TreeNet 

Salford  Systems 

JIAT 

ODASA-CE 

TreePlan 

TreePlan  Software 

IMP 

SAS 

TreePlan  Toolkit 

TreePlan  Software 

KNITRO 

Ziena  Optimization 

EEC 

VisLab 

MIT 

Logical  Decisions 
for  Windows 

Logical  Decisions 

VisualDOC 

Vanderplaats  Rc&D,  Inc. 

Logical  Decisions 
Portfolio 

Logical  Decisions 

Web-HIPRE 

SAL,  Aalto  University 

MapleSim 

MARS 

Maplesoft 

Salford  Systems 

WSTAT 

PEO-GCS 

Notes:  AAMODAT  =  Armament  Analysis  Multiple  Objectives  Decision  Analysis  Tool;  ARDEC  =  US 
Army  Research,  Development  and  Engineering  Center;  ASDL  =  Aerospace  Systems  Design  Laboratory; 
ASEC  =  Advanced  Systems  Engineering  Capability;  ATSV  =  ARL  Trade  Spaec  Visualizer;  CART  =  Clas¬ 
sification  and  Regression  Trees;  CPAT  =  Capability  Portfolio  Analysis  Tool;  DAKOTA  =  Design  Analysis 
toolKit  for  Optimization  and  Terascale  Applications;  FACT  =  Framework  for  Assessing  Cost  and  Technol¬ 
ogy;  GRIPS  =  Genetic  Resources  for  Innovation  and  Problem  Solving;  GTRI  =  Georgia  Tech  Research  In¬ 
stitute;  JIAT  =  Joint  Integrated  Analysis  Tool;  LMI  =  Logistics  Management  Institute;  MARS  =  Multivariate 
Adaptive  Regression  Splines;  MIT  =  Massachusetts  Institute  of  Technology;  NASA  =  National  Aeronau¬ 
tics  and  Space  Administration;  ODASA-CE  =  Office  of  the  Deputy  Assistant  Secretary  of  the  Army  for 
Cost  and  Economics;  PEO-GCS  =  Program  Executive  Office  Ground  Combat  Systems;  PSU-ARL  =  Penn¬ 
sylvania  State  University  Applied  Research  Laboratory;  RICH  =  Rank  Inclusion  in  Criteria  Hierarchies; 
ROSETTA  =  Relational-Oriented  Systems  Engineering  and  Technology  Tradeoff  Analysis;  RPM  =  Robust 
Portfolio  Management;  SAE  =  Systems  Analysis  and  Experimentation;  SAL  =  Systems  Analysis  Labora¬ 
tory;  SCAP  =  System  Capabilities  Analytic  Process;  SLAD  =  Survivability/Lethality  Analysis  Directorate; 
SPIDR  =  Systems  Platform  for  Integrated  Design  in  Realtime;  TARDEC  =  US  Army  Tank  Automotive  Re¬ 
search,  Development  and  Engineering  Center;  TIES  =  Technology  Identification,  Evaluation,  and  Selection; 
TRACER  =  Tradespace  Analysis  for  Capabilities,  Effectiveness,  and  Resources;  WSTAT  =  Whole  System 
Trades  Analysis  Tool 
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4.2  TSE  Tool  Reduction 


After  identifying  a  large  number  of  tools  for  initial  consideration,  the  next  step 
was  to  reduce  the  set  to  a  manageable  size,  knowing  that  the  subsequent  tasks 
would  involve  identifying  attributes  and  then  mapping  tools  to  those  attributes  in 
an  attribute-by-tool  sized  matrix.  A  set  of  keep-or-reject  criteria  was  developed. 
The  first  criterion  was  a  gate:  if  the  tool  was  in  use  by  any  of  the  ERS  DP  teams, 
then  it  was  kept  for  consideration  in  the  present  study,  and  if  not,  then  it  entered 
into  a  set  of  filter  criteria.  The  first  filter  criterion  was  whether  the  tool  or  devel¬ 
oper  was  still  available;  in  several  cases  an  Internet  search  for  the  tool  or  developer 
returned  limited  results  or  no  web  page.  The  next  filter  criterion  was  whether  the 
tool  was  included  as  part  of  a  larger  suite  of  tools;  in  several  cases  individual  tools 
were  rolled  into  a  larger  suite  by  the  developer,  in  which  case  the  suite  was  assessed 
for  evaluation  as  opposed  to  the  individual  tools.  The  final  filter  criterion  was  if  the 
tool  was  specific  to  a  domain  outside  of  system  design  and  analysis  (e.g.,  medical 
or  service  industry);  tools  with  a  niche  function  were  omitted  to  focus  on  tools  that 
encompassed  a  broad  TSE  process.  This  initial  filtering  reduced  the  tool  set  by  38. 

The  remaining  43  tools  were  reassessed  to  determine  if  they  should  be  evaluated  in 
the  current  effort  or  deferred  to  a  follow-on  effort.  The  first  filter  criterion  at  this 
level  was  if  the  tool’s  development  or  use  was  supported  by  any  known  DOD  TSE 
effort,  or  if  it  was  presently  receiving  DOD  funding  for  continued  development 
and  application;  if  so,  then  it  was  kept  for  current  consideration.  The  next  filter 
criterion  was  if  the  tool  received  high  ranking  and/or  positive  feedback  from  other 
tool  surveys;  several  tools  showed  consistent  or  rising  popularity,  which  the  authors 
assessed  as  indicative  of  high  user  satisfaction.  The  final  filter  criterion  was  if  the 
authors  had  any  familiarity  with  the  tool  whatsoever;  if  none,  then  the  tool  was 
deferred  for  assessment  since  the  only  other  information  available  was  developer 
marketing  and  advertising  literature,  and  the  authors  felt  it  inappropriate  to  rely  on 
secondary  knowledge  to  complete  the  initial  assessment.  Using  these  criteria  the 
subset  of  43  tools  was  reduced  to  13,  shown  in  Table  2. 

Regardless  of  which  tools  were  selected  for  immediate  evaluation,  the  recommen¬ 
dation  is  for  the  43  tools  to  be  assessed  against  the  forthcoming  attributes,  either  by 
a  user  or  the  developer,  because  of  the  TSE  capability  they  potentially  bring  through 
individual  use  or  in  concert  with  other  tools. 
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Table  2  TSE  tools  and  processes  for  immediate  evaluation 


Tool/Process 

Developer 

AAMODAT 

ARDEC 

ASEC 

TARDEC 

ATSV 

PSU-ARL 

CPAT 

PEO-GCS 

Excel 

Microsoft 

EACT 

GTRI 

JMP 

SAS 

MATLAB 

Mathworks 

ModelCenter 

Phoenix  Integration 

OpenMDAO 

NASA 

R 

R  Eoundation 

TIES 

ASDL,  Georgia  Tech 

WSTAT 

PEO-GCS 

Notes:  AAMODAT  =  Armament  Analysis  Multiple  Objectives  Decision  Analysis 
Tool;  ARDEC  =  US  Army  Research,  Development  and  Engineering  Center;  ASDL 
=  Aerospace  Systems  Design  Laboratory;  ASEC  =  Advanced  Systems  Engineer¬ 
ing  Capability;  ATSV  =  ARE  Trade  Spaec  Visualizer;  CPAT  =  Capability  Portfolio 
Analysis  Tool;  PACT  =  Framework  for  Assessing  Cost  and  Technology;  GTRI  = 
Georgia  Tech  Research  Institute;  NASA  =  National  Aeronautics  and  Space  Ad¬ 
ministration;  PSU-ARL  =  Pennsylvania  State  University  Applied  Research  Labora¬ 
tory;  PEO-GCS  =  Program  Executive  Office  Ground  Combat  Systems;  TARDEC  = 
US  Army  Tank  Automotive  Research,  Development  and  Engineering  Center;  TIES 
=  Technology  Identification,  Evaluation,  and  Selection;  WSTAT  =  Whole  System 
Trades  Analysis  Tool 


The  full  set  of  rationale  for  keeping  or  rejeeting  a  tool  for  the  present  or  follow-on 
assessment  is  provided  in  Appendix  B. 

4.3  TSE  Tool  Functionality 

After  applying  the  filter  eriteria,  the  entire  landseape  of  tools  was  revisited  to  see 
if  eaeh  tool  eould  be  assigned  a  primary  funetion.  The  purpose  for  this  was  2-fold: 
determine  how  many  deeision  analysis  tools  had  been  earried  over  for  assessment, 
and  address  the  observation  that  all  of  the  tools  appeared  to  be  meeting  a  repet¬ 
itive  set  of  funetions.  The  apparent  funetions  are  shown  in  Table  3,  along  with  a 
brief  deseription  and  the  number  of  tools  that  have  the  funetion  as  their  primary  and 
seeondary.  As  expeeted  based  on  relying  on  the  OR/MS  DAS  tool  surveys  for  tool 
input,  deeision  analysis  tools  made  up  a  large  portion  of  the  initial  tool  landseape 
for  this  assessment.  It  should  be  noted  that  although  a  multidiseiplinary  optimiza¬ 
tion  (MDO)  funetion  is  speeified  in  Table  3,  ERS  is  not  focused  on  identifying  an 
optimum  system  but  rather  on  assessing  resilience  across  multiple  use  cases. 
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Table  3  TSE  tool  functionality 


Function 

Short  Description  of  Function 

Tools  with 
Primary 

Tools  with 
Secondary^ 

Capturing  Value 

Using  weightings,  value  measures, 
surveys,  or  other  techniques  to  capture 
user-perceived  value 

3 

9 

MDO 

Linking  multidisciplinary  models  for  the 
purpose  of  optimizing  a  constrained  or 
unconstrained  objective  function 

12 

5 

Statistical  Data 
Analysis 

Identifying  trends  or  patterns  in  data, 
and  developing  models  to  help  forecast 
outputs  based  on  inputs 

25 

15 

Visualization 

Displaying  information  graphically  with 
static  or  dynamic  charts  and  tables 

4 

19 

Decision  Analysis 

Evaluating  sets  of  alternatives  against 
preferences  on  outcomes 

25 

17 

Project/Process 
Portfolio 
Management  and 
Simulation 

Assessing  how  much  corporate  value 
can  be  achieved  from  project  portfolio 
combinations,  resource  allocation 
options,  and  scheduling 

12 

2 

Notes:  MDO  =  multidisciplinary  optimization 
“Not  all  tools  have  an  explicit  secondary  function. 


While  the  tools  eonveniently  fell  under  these  6  primary  function  categories,  they 
should  not  be  considered  to  perform  only  these  functions.  Tools  can  readily  be 
differentiated  from  each  other  when  considering  lower-level  functions  such  as  pro¬ 
cess  simulation,  data  mining,  regression,  decision  trees,  brainstorming,  Monte  Carlo 
simulation,  sensitivity  analysis,  and  requirements  analysis. 

4.3.1  Capturing  Value 

Quantifying  the  value  of  system  attributes,  metrics,  and  goals  based  upon  user  per¬ 
spectives  allows  for  mathematical  processes  to  logically  operate  upon  these  val¬ 
ues.  Quantified  values  help  set  the  basis  for  decision  analyses  at  the  conclusion 
of  TSE.  Capturing  perceived  value  enables  multiple,  potentially  competing,  stake¬ 
holder  preferences  to  be  modeled.  The  purpose  of  capturing  value  is  not  to  over¬ 
constrain  the  capability  space  and  find  the  "gold  plated"  system  that  maximizes 
value  for  all  stakeholders  simultaneously;  understanding  where  stakeholders  place 
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value  illuminates  the  tradeoffs  that  must  be  addressed  and  reveals  capability  space 
that  may  be  a  best  compromise  for  all.  Tools  that  facilitate  capturing  value  do  so 
by  eliciting  weights,  typically  in  such  a  way  that  quantitative  or  qualitative  surveys 
address  mathematical  independence  conditions.  Other  methods  and  tools  involve 
mapping  the  logical  flow  of  ideas  or  building  scenarios  and  storyboards. 

4.3.2  Multidisciplinary  Optimization  (MDO) 

Typical  systems  engineering  practices  call  for  analyses  within  multiple  disciplines. 
What  might  be  an  optimal  solution  to  an  aerodynamicist  could  very  well  be  the 
poorest  solution  to  the  structures  engineer.  It  is  common  for  designs  to  pass  from 
one  engineering  group  to  another,  making  modifications  along  the  way  in  an  itera¬ 
tive  process.  MDO  is  a  technique  whose  goal  is  to  streamline  this  iterative  process 
and  apply  optimization  algorithms  while  maintaining  input  from  subject  matter  ex¬ 
perts.  Multiple  tools  such  as  computational  fluid  dynamics,  computational  structural 
dynamics,  and  statistical  analysis  software  are  linked  together  to  create  a  central¬ 
ized  global  system  analysis  model  where  results  are  automatically  passed  from  each 
analysis  to  the  next  with  little-to-no  user  interaction.  MDO  has  the  capability  of 
multifidelity  analyses  in  that  it  is  at  the  engineer’s  discretion  as  to  how  many  anal¬ 
yses  are  used,  and  their  depth  of  detail.  This  allows  the  advantages  of  MDO  to  be 
applied  in  the  preliminary  concept  design  phase,  all  the  way  down  to  component 
level  design.  With  MDO,  the  tradespace  can  be  generated  and  explored  repeatedly 
and  at  varying  levels  of  fidelity  to  drive  the  system  development  process  toward 
the  optimal  solution  in  an  efficient,  integrated  manner.  It  should  be  noted  that  al¬ 
though  an  MDO  function  is  specified  in  Table  3,  ERS  is  not  focused  on  identifying 
an  optimum  system  but  rather  on  assessing  resilience  across  multiple  use  cases. 

4.3.3  Statistical  Data  Analysis 

Statistical  data  analysis  forms  the  core  of  TSE.  Decision  makers  at  multiple  hi¬ 
erarchical  levels  desire  to  find  relationships  in  their  data  to  draw  conclusions  and 
help  predict  future  outcomes  under  uncertainty.  Tasks  such  as  design  of  experi¬ 
ments  (DOE),  surrogate  modeling,  Monte  Carlo  simulations,  machine  learning,  and 
pattern  recognition  are  used  to  reveal  known,  and  discover  unknown,  relationships 
in  data.  These  techniques  also  provide  the  basis  for  certain  data  visualizations  that 
can  aid  decision  makers  in  finding  trends  and  points  of  interest  within  data,  which 
would  not  otherwise  have  been  found. 
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4.3.4  Visualization 

TSE  would  be  incomplete  without  communicating  insights  of  a  complex  data  set  in 
an  intuitive  and  graphical  manner.  Data  visualizations  incite  discovery  of  informa¬ 
tion  such  as  trends,  inflection  or  saddle  points,  and  statistical  properties.  The  ease 
at  which  useful  observations  can  be  made,  without  mathematically  heavy  statisti¬ 
cal  analysis,  presents  potential  to  uncover  new  relations  and  can  be  used  to  drive 
further  analyses.  With  a  wide  range  of  tools  of  varying  customizability  and  price, 
an  engineer  would  be  remiss  to  omit  a  data  visualization  tool  from  their  analysis 
package  as  it  can  be  used  to  support  other  TSE  functions  such  as  decision  analysis. 

4.3.5  Decision  Analysis 

The  primary  reason  for  comparing  alternatives  is  to  eventually  make  a  selection 
based  on  the  information  at  hand.  Alternatives  are  assessed  for  how  well  they  per¬ 
form  in  satisfying  an  overall  preference.  The  process  of  decision  analysis  relies  on 
stakeholder  input,  alternative  analysis,  and  an  overall  comparison  of  how  well  alter¬ 
natives  meet  objectives.  Decision  analyses  can  consist  of  simple  forms  of  weighing 
pros  and  cons  for  each  alternative,  or  be  as  complex  as  necessary  to  reach  a  decision 
with  as  little  ambiguity  as  possible.  A  formal  decision  analysis  process  can  show  the 
uncertainty  of  a  solution  and  its  probability  to  meet  goals  by  using  single-  or  mul¬ 
tiobjective  utility  functions  that  quantify  the  success  of  a  given  choice.  Decision 
makers  commonly  rely  on  visualizations  and  statistical  analysis  results  to  make  an 
informed  decision,  making  presentation  of  analysis  results  critical. 

4.3.6  Project/Process  Portfolio  Management  and  Simulation 

Driven  by  the  need  to  maximize  corporate  value  under  resource  and  schedule  con¬ 
straints,  companies  often  rely  on  project  portfolio  management  tools  to  design  and 
simulate  portfolio  investment  strategies.  Included  in  this  category  are  tools  that  per¬ 
form  process  flow  optimization  (e.g.,  service  industry  modeling  or  assembly  line 
throughout).  Often,  these  tools  rely  on  a  structured  value  assessment  process  where 
corporate  objectives  are  defined  and  quantified,  alternative  outcomes  are  assigned  a 
probability  of  occurrence,  portfolio  elements  are  mapped  to  corporate  value  func¬ 
tions,  and  stochastic  simulations  reveal  probability  of  success. 

Interestingly,  the  categories  identified  here  through  observation  closely  matched 
the  categories  developed  for  a  workshop  on  Data  Driven  Tradespace  Exploration 
and  Analysis  organized  by  ARE  and  hosted  by  the  Potomac  Institute  for  Policy 
Studies^"^,  shown  in  Table  4. 
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Table  4  Tradespace  workshop  technical  areas  and  tool  functions 


Tradespace  Workshop  Technical  Area 

TSE  Tool  Function  Analog 

Broaden,  Populate,  Manage 

N/A 

Link 

MDO 

Search,  Explore,  Analyze 

Statistical  analysis,  visualization 

Act 

Capturing  value,  decision  analysis 

Notes:  MDO  =  multidisciplinary  optimization;  N/A  =  not  applicable 


4.4  TSE  Tool  Attributes 


Continuing  with  the  reduced  set  of  tools,  the  next  task  was  to  identify  attributes  that 
constitute  a  TSE  tool  and  define  the  functionality  desired  in  such  a  tool.  Referring 
back  to  the  sources  in  Section  4.1,  an  initial  set  of  25  attributes  was  developed, 
shown  in  Fig.  2. 


Fig.  2  Initial  set  of  TSE  tool  attributes 


Other  sources  were  then  reviewed  in  the  areas  of  visualization,  optimization,  and 
value^^^^°  and  the  industry  surveys^^^  were  more  thoroughly  reviewed  given  their 
popularity  and  the  quality  in  their  results  based  on  the  assumed  standardization  of 
their  questions  and  corresponding  attributes. 

An  interesting  discovery  was  made  when  attempting  to  validate  the  assumed  stan¬ 
dardization.  The  OR/MS  Today  DAS  tool  surveys  are  set  up  as  online  question¬ 
naires.  A  hyperlink  is  provided  for  tool  vendors  and  developers  to  describe  their 
software  by  entering  in  yes  ("y"),  no  ("n"),  or  a  limited  textual  description.  The 
2010  DAS  survey^  did  not  provide  rationale  or  descriptions  of  the  survey  questions. 
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There  was  also  no  indication  of  the  provenance  of  the  questions,  other  than  a  short 
statement  about  the  industries  from  which  new  questions  were  pulled:  "For  this 
year’s  [2010]  survey... questions  based  on  the  input  of  decision  analysis  practition¬ 
ers  from  government,  airline,  oil  and  gas  and  pharmaceutical  sectors  were  added." 
As  an  aside,  4  types  of  DAS  materialized  from  the  results  of  the  2010  DAS  tool 
survey  results^,  which  also  closely  resemble  the  functions  identified  in  Table  4: 

•  Problem  structuring  and  brainstorming 

•  Multiobjective  decision  analysis,  multicriteria  decision  analysis,  analytic  hi¬ 
erarchy  process,  and  expert  systems  used  to  prioritize  a  list  of  existing  options 

•  Single  attribute  decision  making  with  uncertainty 

•  Uncertainties  and  probabilistic  assessments,  plus  forecasting,  optimization, 
and  operations  research 

In  the  2012  DAS  tool  survey^,  the  same  questions  were  provided  to  the  vendors 
who  had  responded  in  previous  years,  as  well  as  to  vendors  familiar  to  the  survey 
moderator — not  unlike  what  was  done  in  this  assessment  when  filtering  for  tools  to 
include  for  immediate  evaluation. 

An  electronic  mail  inquiry  sent  to  the  2010  and  2012  DAS  tool  survey  moderators 
revealed  that  the  origin  of  the  survey  questions  most  likely  traces  back  to  a  sin¬ 
gle  individual,  and  there  is  no  recorded  pedigree  on  the  questions  or  attributes,  nor 
are  formal,  objective,  unambiguous  definitions  available.  Further,  each  moderator 
responsible  with  delivering  the  survey  in  a  given  year  is  permitted  to  update  at¬ 
tributes  as  they  see  necessary  (e.g.,  when  technology  advances  result  in  a  question 
or  attribute  being  no  longer  applicable).  Lastly,  the  tool  vendors  are  permitted  to 
"self-assess"  their  product’s  capability  against  the  attributes,  meaning  the  attributes 
are  open  to  interpretation  by  those  completing  the  survey. 

Based  on  the  above  description  of  the  DAS  tool  surveys,  the  authors  of  this  report 
have  recommended  to  the  Institute  for  Operations  Research  and  the  Management 
Sciences  that  the  2014  OR/MS  Today  DAS  survey  moderator  consider  standardiz¬ 
ing  the  questions  and  attributes  by  developing  unambiguous  definitions,  possibly 
using  the  work  performed  here  as  a  starting  point. 
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After  revisiting  the  initial  TSE  tool  attributes,  the  set  increased  to  91,  practically 
eliminating  the  ability  to  perform  a  VET  approach  at  this  stage.  The  attributes  were 
then  grouped  into  common,  higher  level  affinity  categories,  which  are  shown  in 
Table  5  with  a  short  description  and  the  number  of  attributes  per  category.  It  should 
be  noted  that  these  categories  are  a  mix  of  quantitative  and  qualitative,  including 
binary  (e.g.,  y  or  n),  enumerated  (e.g.,  small,  medium,  large),  textual,  currency,  and 
integer. 


Table  5  Top-level  TSE  tool  attributes 


Category 

Short  Description  of  Category 

Number  of 
Attributes 

Class 

Is  the  product  better  classified  as  a  tool,  or  a  process 

2 

Usage 

Industries,  market  segments,  or  applications  the  is  tool 
used  in 

3 

Operating  systems 

On  which  operating  systems  can  the  product  operate 

6 

Pricing 

What  is  the  price,  or  pricing  structure,  of  the  product 

4 

Training  classes 
offered 

What  training  options  available  for  the  product 

4 

Training/experience 

needed 

Knowledge  level  of  subject  matter  to  effectively  use 
the  product 

3 

Software  attributes 

Limitations,  expandability,  and  help  options 

8 

Applications 

Decision  making,  preference,  risk,  and  uncertainty 
capabilities 

8 

Software  features 

Meaningful  functions  and  user  experience 

34 

Decision  algorithm 
implemented 

How  does  the  product  rank  order  alternatives 

3 

Availability  of 
graphical  elicitation 
techniques 

How  does  the  product  capture  values  and  preferences 

9 

Types  of  output 
display 

How  does  the  product  present  results  for  manipulation 

7 

The  full  set  of  attribute  categories  and  their  corresponding  descriptions  and  rationale 
are  provided  in  Appendix  C. 
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A  key  capability  worth  noting  is  the  expansion  of  a  tool’s  functionality  through  cus¬ 
tom  coding/programming/scripting,  or  integrating  code  developed  by  others.  The 
significance  of  this  capability  can  readily  be  appreciated  when  understanding  that  a 
tool  that  supports  customization  through  programming  can  be  modified  to  include 
any  conceivable  function  if  the  user  has  the  time  and  knowledge  to  perform  the 
programming.  A  familiar  example  is  the  ubiquitous  Microsoft  Excel^^  spreadsheet 
software.  The  built-in  functionality  of  this  tool  is  primarily  for  tabulation  and  chart¬ 
ing  of  data,  statistical  analysis,  and  numerical  solving.  However,  if  familiar  with  the 
Visual  Basic  for  Applications  (VBA)  language,  the  user  can  expand  and  customize 
Excel  to  perform  any  of  the  primary  and  lower-level  functions  described  above. 

4.5  TSE  Tool  Survey 

With  the  tools  and  attributes  identified,  the  next  task  was  to  essentially  self-take  the 
survey  for  the  13  tools  in  Table  2.  Based  on  author  knowledge  of  the  tools  and  how 
well  they  met  the  attributes,  the  initial  survey  was  completed  with  results  provided 
in  Appendix  D.  Even  with  detailed  knowledge  and  experience  on  a  majority  of 
these  tools,  gaps  still  existed  in  the  assessment  related  to  pricing  as  well  as  some 
of  the  attributes  pulled  in  from  the  DAS  Surveys.  The  latter  is  driven  by  attribute 
ambiguity  and  subjectivity  as  explained  earlier. 

It  is  envisioned  that  the  TSE  Tool  Survey  will  be  fully  populated  by  calling  upon 
experienced  users,  and  possibly  developers,  of  the  identified  tools.  To  do  this,  the 
tool  attributes  query  should  be  formatted  as  a  questionnaire.  The  questions  should 
follow  survey  best  practices,  being  as  specific  and  concise  as  possible  to  remove 
ambiguity  and  bias. 

An  important  conclusion  from  this  survey  can  be  drawn  based  on  the  holistic  view 
of  TSE  as  described  thus  far:  having  a  valid  set  of  attributes,  and  an  understanding 
of  how  a  cross  section  of  tools  can  satisfy  them,  is  insufficient.  What  is  now  needed 
is  a  deeper  understanding  of  how  these  tools  are  used  and,  more  importantly,  how 
they  can  be  used  when  performing  TSE  in  support  of  the  decision  analysis  process. 
Gaining  this  understanding  will  enable  users  to  better  assess  if  they  possess  the 
appropriate  tools  for  the  TSE  steps  that  they  need  to  perform,  as  well  as  which  TSE 
steps  are  capable  of  being  performed  given  the  tools  in  the  user’s  possession. 
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5.  Tradespace  Analysis  Decision  Classification 


This  portion  of  the  effort  involved  looking  aeross  several  prominent  TSE  researeh 
aetivities  to  determine  if  there  exists  a  eommon,  best  praetiee  proeess  for  perform¬ 
ing  TSE.  Erom  these  efforts,  a  eonsolidated  proeess  was  outlined  and  steps  eom- 
pared  aeross  the  ERS  DPs  and  2  other  aetivities  of  interest  to  determine  whether 
these  efforts  were  eonsistently  performing  a  TSE  proeess  and,  if  not,  whieh  steps 
were  being  omitted.  A  reeommendation  is  offered  for  mapping  TSE  tools  to  at¬ 
tributes  and  to  proeess  steps. 

5.1  TSE  Process  Audit 

Thus  far,  tools  were  identified  that  may  be  useful  in  performing  TSE,  doeumenting 
their  main  funetions,  and  eolleeting  their  defining  attributes.  However,  these  tools 
are  intended  for  use  aeross  the  system  development  life  eyele,  in  the  eontext  of 
a  proeess  for  performing  TSE  and  making  deeisions.  Therefore,  it  is  important  to 
understand  how  these  tools  ean  be  intentionally  used  to  support  a  eonsistent  TSE 
proeess. 

TSE  researeh  aetivities  aeross  government  and  aeademia  were  investigated  to  look 
for  eommonality  in  proeess  exeeution,  shown  in  Table  6.  Additionally,  TSE  projeets 
under  study  within  the  government  (Table  7)  were  investigated  to  better  understand 
what  steps  were  being  exeeuted  aeross  the  full  TSE  aetivity.  Detailed  deseriptions 
of  the  three  DPs  are  provided  in  Appendix  E.  Detailed  deseriptions  of  Capability 
Portfolio  Analysis  Tool  (CPAT)  and  Eramework  for  Assessing  Cost  and  Teehnology 
(EACT)  are  provided  in  Appendix  E  and  Appendix  G,  respeetively.  Due  to  limited 
availability  of  information,  DPS  was  not  investigated. 
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Table  6  Prominent  TSE  research  activities 


Organization 

TSE  Activity 

Reference 

Georgia  Tech 
ASDL 

Integrated  Reconfigurable  Intelligent  Systems  Design 
Process 

[22] 

DSTA 

Tradespace  Exploration  for  Military  Simulations 

[23] 

MIT 

Responsive  Systems  Comparison  Method 

[24] 

USMA 

Expanding  the  Tradespace 

[25] 

PSU-ARL 

Visual  Steering  Commands  for  Tradespace  Exploration 

[26] 

NRL 

Goal-Oriented  Computational  Steering 

[27] 

NASA 

Visualizing  Requirements  and  Risk 

[28] 

Notes:  ASDL  =  Aerospace  Systems  Design  Laboratory;  DSTA  =  Singapore  Defence  Science  & 
Technology  Agency;  MIT  =  Massachusetts  Institute  of  Technology;  NASA  =  National  Aeronau¬ 
tics  and  Space  Administration;  NRL  =  US  Naval  Research  Laboratory;  PSU-ARL  =  Pennsylva¬ 
nia  State  University  Applied  Research  Laboratory;  USMA  =  United  States  Military  Academy 


Table  7  Projects  investigated  for  TSE  best  practice 


Organization 

TSE  Activity 

Reference 

AEMC 

DPI:  Eixed  wing  aircraft  (Global  airliner  C-X) 

[8] 

NSWC 

DP2;  Ship  design  (Littoral  combat  ship) 

[9,10] 

ERDC 

DP3;  Sensor  systems 

PEO-GCS 

CPAT 

[29-31] 

GTRI 

PACT 

[32,33] 

Notes:  AFMC  =  Air  Force  Materiel  Command;  CPAT  =  Capability  Portfolio  Analysis  Tool; 
DP  =  demonstration  project;  ERDC  =  US  Army  Engineer  Research  and  Development  Center; 
EACT  =  Framework  for  Assessing  Cost  and  Technology;  GTRI  =  Georgia  Tech  Research  Insti¬ 
tute;  NSWC  =  Naval  Surface  Warfare  Center;  PEO-GCS  =  Program  Executive  Office  Ground 
Combat  Systems 


5.2  TSE  Process  Steps  and  Coverage 

From  the  research  activities  in  Table  6  and  pilot  projects  in  Table  7,  a  set  of  12 
common  steps  emerged,  as  shown  in  Table  8.  It  is  assumed  that  these  steps  can 
be  considered  "TSE  best  common  practice",  although  this  assumption  is  not  tested 
here.  The  steps  are  not  presented  in  flowchart  format  to  stress  that  a  formal  TSE 
process  is  not  being  proposed  but  rather  existing  processes  consolidated. 
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Table  8  TSE  best  common  practice  steps 


Step  Description 

1  Determine  mission  scenario(s)  and  their  requirements,  and  keep  them  open  as 
long  as  possible 

2  Identify  set  of  operational  performance  characteristics  and  high  level  system 
design  variables  that  impact  operational  requirements 

3  Apply  operational  engagement  models  against  various  mission  scenarios  and 
threats  to  identify  requirements,  measures  of  performance,  measures  of 
effectiveness,  and  other  performance  metrics 

4  Expert  knowledge  teams  determine  values  of  measures  for  given  mission 
scenarios  and  requirements 

5  Break  down  stakeholder  values  into  roles,  attributes,  and  specific  tasks 

6  Generate  alternatives  that  meet  requirements  and  constraints,  and  map 
stakeholder  values  to  system  design  variables  using  scalable  multiphysics 
based  modeling  design  tools 

7  Create  reduced-order  surrogate  models  to  show  iterative  ability  of  adjusting 
scenarios  and  requirements  to  physical  feasibility 

8  Qualitatively  or  quantitatively  rank  how  alternatives  meet  measures 

9  Perform  a  life  cycle  cost  estimate  and  life  cycle  schedule  analysis  of  the 
system 

10  Perform  an  optimization  study  to  determine  the  optimum  feasible  space  that 
meets  all  constraints  and  for  each  course  of  action 

1 1  Determine  courses  of  action  based  on  optimal  feasible  space  and  perform 
postanalysis  studies  (operational  impact  and  gap  analyses) 

12  Perform  case  studies  to  test  for  robustness  and  to  make  sure  that  the 
alternative  solutions  are  resilient  in  changing  operational  environments 


The  TSE  approaches  of  the  ERS  DPs  in  Table  7  were  compared  to  the  steps  in  Table 
8  to  determine  if  these  pilots  were  indeed  good  representations  of  TSE  best  practice. 

The  results  of  this  assessment  are  provided  in  Eigs.  3  and  4.  Shown  across  the  top 
of  the  figures  are  the  steps  from  Table  8.  Shown  down  the  left  side  of  the  figures 
are  the  activities  from  Table  7.  In  the  center  of  the  figures  are  blocks  indicating 
which  tools  are  used  to  perform  the  step  within  each  activity.  In  some  cases  the 
tool  is  not  applicable;  this  indicates  that  external  information  was  provided  or  that 
the  step  was  conducted  internally  to  the  activity  without  reliance  on  an  explicit, 
external  tool.  Adjacent  blocks  showing  the  same  tool  indicate  discrete  steps  within 
the  activity  that  are  using  the  same  tool  (e.g.,  Microsoft  Excel  in  CPAT).  DPS  was 
not  mapped  to  the  TSE  steps  in  this  study,  but  should  be  addressed  in  future  work. 
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Fig.  3  TSE  activities  and  their  mapping  to  TSE  common  steps  1-6 
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time  what-if  scenarios  to  test 
resilience 


Fig.  4  TSE  activities  and  their  mapping  to  TSE  common  steps  7-12 


The  consolidation  and  subsequent  comparison  to  select  projects  revealed  that:  1) 
TSE  is  commonly  performed  in  steps,  as  assumed;  2)  there  exists  no  formal,  singu¬ 
lar,  consistent  process  for  performing  TSE  across  the  organizations  represented  by 
the  activities  in  Table  7;  3)  there  exists  no  singular  step  within  TSE  processes  that 
performs  the  action  of  "exploring"  a  tradespace  (when  such  an  action  is  performed  it 
is  done  so  using  prerequisite  inputs  from  multiple  steps  while  simultaneously  feed¬ 
ing  back,  and  forward,  exploration  results  to  the  decision  analysis  process);  and  4) 
the  evaluated  ERS  DPs  both  omitted  steps  4  (subject  matter  expert  measures)  and  5 
(stakeholder  value). 

Also  from  Eigs.  3  and  4,  it  can  be  seen  that  while  CPAT  does  take  into  account 
value  (step  5),  it  does  not  have  an  established  method  or  tool  for  identifying  oper¬ 
ational  performance  characteristics  and  high-level  design  variables  that  impact  op¬ 
erational  requirements  (step  2);  these  attributes  are  absent  based  on  CPAT’s  role  as 
a  portfolio  modernization  investment  decision-making  tool  as  opposed  to  being  an 
iterative  design  and  optimization  tool  that  links  user  preferences  to  design  capabil¬ 
ities. PACT,  on  the  other  hand,  while  not  allowing  real-time  definition  of  system 
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design  variables  (step  2),  does  map  the  variables  to  system  functions  and  physical 
components  and  defines  their  uncertainty,  with  the  aid  of  a  work  breakdown  struc¬ 
ture  (WBS)  and  random  distribution  assignments.^^  However,  unlike  CPAT,  FACT 
does  not  support  real-time  allocation  of  stakeholder  values  to  roles,  attributes,  and 
specific  tasks  (step  5).  The  two  DPs  do  not  perform  an  optimization  study  as  part  of 
a  discrete  step  (step  10),  but  rather  they  roll  optimization  into  a  process  that  calcu¬ 
lates  alternative  capabilities  while  simultaneously  looking  across  capability  gaps. 

Of  all  of  the  efforts,  FACT  is  the  closest  in  terms  of  completely  performing  the 
best  practice  TSE  steps.  However,  there  is  room  to  improve  the  level  of  interaction 
between  the  users  and  the  tradespace;  in  some  steps  the  inputs  are  generated  within 
FACT  (not  with  a  specific  tool)  and  in  other  steps  FACT  relies  on  static  information 
provided  by  the  user. 

5.3  TSE  Steps,  Tools,  and  Functions 

Consider  that  each  of  the  81  identified  tools,  defined  by  a  subset  of  the  91  identified 
attributes,  gives  users  an  ability  to  perform  many  of  the  12  identified  TSE  best 
practice  steps.  The  fully  populated  TSE  Tool  Survey  will  provide  a  link  between 
TSE  best  practice  steps,  tools,  and  functions,  allowing  users  to  balance  their  TSE 
needs  with  their  organizational  capabilities  and  constraints.  The  survey  will  serve 
as  a  database  that  can  be  used  in  different  ways:  users  can  decide  which  tools  they 
should  invest  in  by  selecting  the  TSE  steps  that  they  desire  to  perform;  or  users 
can  identify  which  TSE  steps  can  be  performed  with  the  tools  currently  available 
within  their  organization.  Knowledge  level,  budget,  and  training  can  serve  as  filters 
in  the  user’s  decision  process.  A  recommendation  for  future  research  is  to  develop 
a  decision  analysis  style  user  interface  for  aiding  users  in  downselecting  the  most 
appropriate  tools  based  on  their  TSE  needs  and  goals. 
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6.  Conclusion  and  Recommendations 


This  effort  used  attributes  of  TSE  tools  and  the  steps  in  a  notional  TSE  proeess 
to  illuminate  relationships  between  tool  eapabilities  and  proeess  steps  to  help  ex¬ 
isting  projeets  within  the  ERS  TSE  Pillar  understand  if  they  are  suffieiently  and 
eonsistently  performing  what  is  eonsidered  to  be  neeessary  TSE  for  ERS.  Although 
a  speeified  (or  default  standard)  formal  TSE  proeess  is  not  in  use  for  DOD  pro¬ 
grams,  this  effort  has  provided  insight  into  a  possible  standard  TSE  proeess  for 
ERS  programs.  The  notional  proeess  requires  further  investigation  to  determine  if 
it  makes  an  improvement  over  eurrent  TSE  proeesses.  Multiple  tradespaee  efforts 
in  government  and  industry  were  reviewed  for  TSE  tools,  TSE  tool  attributes,  and 
TSE  proeess  steps.  A  holistie  view  of  81  eandidate  tradespaee  exploration  tools 
is  provided.  Tools  were  grouped  into  eommon  primary  funetions.  Ninety-one  tool 
attributes  were  identified.  A  survey  template  was  ereated  based  on  an  existing  deei- 
sion  analysis  tool  survey,  and  the  survey  was  populated  for  several  tools. 

Reeommendations  for  future  work  inelude  the  following: 

•  Conduet  a  more  formal  VET  approaeh,  developing  value  measures  for  at¬ 
tributes  and  funetions. 

•  Refine  the  list  of  attributes  and  tools  identified  in  this  effort. 

•  Complete  the  mapping  of  tools  to  attributes  through  a  survey  of  experieneed 
users  and  developers.  Request  rationale  to  support  the  survey  entry.  Possibly 
request  an  ordinal  ranking  of  how  well  eaeh  tool  meets  eaeh  attribute  for 
subjeetive  inputs  to  mimie  the  surveys  refereneed  within  this  paper. 

•  Validate  the  TSE  steps  identified  in  this  effort. 

•  Eneourage  the  Institute  for  Operations  Researeh  and  Management  Seienees 
to  adopt  the  attribute  definitions  developed  here  for  their  next  OR/MS  Today 
DAS  survey. 

•  Develop  a  deeision  analysis  style  user  interfaee  to  make  the  database  interae- 
tive  for  the  user. 
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©RISK  (Palisade)  is  an  add-in  to  Microsoft  Excel  spreadsheet  software  that  per¬ 
forms  project  risk  analysis  using  Monte  Carlo  simulation.  Based  upon  user  inputs, 
outputs,  and  models,  users  can  specify  ranges  and  distributions  on  inputs  as  well  as 
the  number  of  trials.  The  tool  determines  the  probability  of  outcomes  and  presents 
the  results  in  tabular  and  plot  format.  Results  are  commonly  presented  to  users  as 
probability  density  functions  (PDFs)  and  cumulative  distribution  functions  (CDFs). 
Options  for  optimization,  time  series  analysis,  correlations,  and  curve  fits  are  in¬ 
cluded  in  the  tool.  ©RISK  enables  a  user  to  plan  for  probabilistic  outcomes  based 
on  input  conditions  and  process  randomness.  Primary  applications  are  resource  al¬ 
location  and  scheduling. 

lOOOMinds  (lOOOMinds  Ltd)  is  a  multicriteria  decision-making  tool  that  is  ac¬ 
cessed  through  an  Internet  browser.  As  of  the  compilation  and  writing  of  this  re¬ 
port,  the  lOOOMinds  web  page  was  not  operational.  The  lOOOMinds  tool  is  based 
off  of  pairwise  comparisons  and  implements  the  method  known  as  potentially  all 
pairwise  rankings  of  all  possible  alternatives.  Primary  applications  are  prioritiza¬ 
tion  and  resource  allocation,  but  lOOOMinds  can  also  be  used  to  determine  decision 
maker  weightings  on  alternative  attributes. 

Advanced  Systems  Engineering  Capability  (ASEC)  (US  Army  Tank  Automo¬ 
tive  Research,  Development  and  Engineering  Center  [TARDEC])  is  a  web- 
based  integrated  systems  engineering  knowledge  creation  and  capture  framework 
built  on  a  decision  centric  method,  high  quality  data  visualizations,  intuitive  navi¬ 
gation  that  enables  continuous  data  traceability,  real-time  collaboration  and  knowl¬ 
edge  pattern  leverage  supporting  the  entire  system  life  cycle.'  ASEC  is  a  single 
framework  that  enables  systems  engineering  knowledge  producers  and  consumers 
to  create,  capture,  share,  analyze,  and  present  decision  makers  with  systems  en¬ 
gineering  life  cycle  knowledge  to  improve  decision  quality  and  confidence.  The 
government-owned  framework  has  been  architected  to  support  model-based  engi¬ 
neering  and  model-based  systems  engineering  (MBSE)  information.  ASEC  lever¬ 
ages  a  decision  model  as  the  integrative  mechanism  that  provides  context  for  its 
other  system  models,  including:  math/physics,  architecture,  life  cycle  ("ilities"),  and 
roadmap.  ASEC  includes  capabilities  to  capture  stakeholder  needs,  manage  system 
requirements,  perform  decision  making,  and  track  program  risks  and  opportunities. 

'After  the  study  period,  ASEC  was  renamed  to  the  Integrated  Systems  Engineering  Eramework. 
New  capabilities  were  added  that  are  not  included  in  this  report. 
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The  feature  in  the  ASEC  framework  most  applieable  to  the  present  study  is  the 
deeision  analysis  tool.  ASEC  allows  for  the  dynamie  and  visual  analysis  of  alterna¬ 
tives  by  linking  requirements,  deeisions,  alternatives,  and  eapabilities.  The  deeision 
analysis  tool  supports  the  analysis  of  alternatives  proeess  ineluding  seoring  of  al¬ 
ternatives  (solutions)  against  eriteria,  eomparing  the  relative  effeetiveness  of  these 
alternatives  (through  spider,  tornado  and  trade-off  eharts)  and  eapturing  the  selee- 
tion/rejeetion  rationale  for  eaeh  alternative. 

AgenaRisk  (Agena  Limited)  uses  Bayesian  networks  to  perform  diagnostie  and 
predietive  reasoning  about  uneertainty  on  stoehastie  models.  Monte  Carlo  simu¬ 
lations  ean  be  run  on  user-speeified  probability  tables.  Outputs  inelude  tables  and 
plots  of  distributions  that  allow  users  to  visualize  the  probability  of  outeomes  in 
PDEs  and  CDEs.  Applieations  inelude  business  risk,  strategie  planning,  proeure- 
ment  risk,  safety,  and  reliability. 

Analytica  (Lumina  Decision  Systems)  is  a  graphieal  modeling  environment  that 
uses  influenee  diagrams  to  explain  how  variables  interaet.  The  graphieal  model¬ 
ing  teehnique  is  meant  to  eliminate  eonfusion  or  hidden  links  that  would  typieally 
reside  in  spreadsheet-based  models.  Users  ean  analyze  risk  and  uneertainty  with 
Monte  Carlo  simulation,  perform  optimization,  and  perform  data  analysis.  Users 
ean  expand  the  eapability  through  existing  libraries,  or  eode  their  own  funetions  in 
the  C  programming  language.  Analytiea  has  been  applied  to  engineering,  business 
modeling,  pharmaeeutieals,  finaneial  planning,  and  more.  Eumina  offers  training, 
or  ean  visit  a  work  site  for  more  eustomized  training. 

Analytics  (SAS)  is  a  commercial  grouping  of  software  from  SAS  covering  predic¬ 
tive  analytics  and  data  mining,  visual  analytics,  forecasting  and  economics,  oper¬ 
ations  research  and  optimization,  model  management  and  monitoring,  quality  im¬ 
provement,  statistics  and  text  analytics.  SAS  proposes  different  software  suites,  in¬ 
dividual  programs,  and  program  plug-ins  for  each  of  these  areas.  Each  software 
solution  from  SAS  caters  toward  different  industry  fields,  levels  of  expertise,  and 
analysis  goals  of  the  user.  The  user  community  includes  groups  from  local  to  inter¬ 
national  levels  that  are  often  supported  by  SAS  via  workshops  and  conferences. 

ARL  Trade  Space  Visualizer  (ATSV)  (Pennsylvania  State  University  Applied 
Research  Laboratory  [PSU-ARL])  is  a  visualization  tool  designed  in  Java  to  help 
users  interact  and  dynamically  explore  complex  tradespaces.  ATSV  can  generate 
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scatter  plots,  scatter  matrices,  histograms,  3-D  glyph  plots,  and  Pareto  frontiers  that 
can  be  dynamically  filtered  and  selected  to  implement  a  visual  steering  command, 
which  allows  a  user  to  narrow  a  search  to  a  chosen  localized  region  of  data.  As  of 
2010,  Penn  State’s  ARL  has  made  ATSV  freely  available  for  anyone  to  download 
and  use.  Some  capabilities  of  this  tool  have  been  licensed  to  Phoenix  Integration 
for  use  in  the  VisualizationPak  in  their  ModelCenter  software. 

Armament  Analysis  Multiple  Objectives  Decision  Analysis  Tool  (AAMODAT) 
(US  Army  Armament  Research,  Development  and  Engineering  Center  [ARDEC]) 

was  created  by  the  System  Decision  Analysis  Group  of  the  US  Army  Armament 
Research,  Development  and  Engineering  Center  (ARDEC)  in  an  effort  to  compare 
multiple  systems  across  5  categories:  unit  cost,  operations  and  support  (O&S)  cost, 
performance,  schedule,  and  growth  potential.  AAMODAT  runs  within  Microsoft 
Excel  spreadsheet  software  and  is  designed  to  walk  the  user  through  a  value-focused 
thinking  (VET)  approach.  AAMODAT  requires  user  inputs  for  objectives,  value 
functions,  swing  weights,  and  priority  weightings.  Users  define  their  systems  us¬ 
ing  quantitative  values  for  each  system  attribute,  along  with  PDEs  to  capture  un¬ 
certainty  in  system  capability.  The  5  output  categories  can  be  plotted  in  various 
ways,  or  users  can  display  value  functions  on  a  radar  chart.  Sensitivities  can  also  be 
performed  to  determine  the  robustness  of  the  analysis  outcome.  AAMODAT  uses 
Palisade’s  ©RISK  tool  to  conduct  Monte  Carlo  simulations. 

Berkeley  Madonna  (UC  Berkeley)  is  a  differential  equation  solver  developed  un¬ 
der  funding  from  the  National  Science  Eoundation  and  National  Institutes  of  Health. 
The  tool  can  perform  root  finding,  curve  fitting,  sensitivity  analysis,  and  optimiza¬ 
tion.  Outputs  include  tables  and  plots.  Currently  written  in  C,  the  developers  have 
moved  to  a  Java  user  interface  with  a  C  execution  engine,  and  are  soon  including 
Python  scripting  capability. 

Capability  Portfolio  Analysis  Tool  (CPAT)  (Program  Executive  Office  Ground 
Combat  Systems  [PEO-GCS])  was  developed  to  provide  a  formal,  structured  pro¬ 
cess  to  address  the  difficulty  associated  with  the  decision  analysis  process  within 
fleet  modernization.  CPAT  follows  a  VET  approach,  which  involves  decomposing 
weighted  roles  into  functions  and  then  using  a  mixed  integer  linear  programming 
optimization  model  to  develop  portfolios  of  systems  that  maximize  fleet  value  over 
time  while  meeting  cost  and  schedule  constraints. 
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Developed  in  VB.NET,  CPAT  helps  establish  the  best  mix  of  existing  vehicles  for  a 
particular  mission  based  on  performance  (how  well  a  vehicle’s  performance  meets 
weighted  attributes),  cost,  and  schedule  constraints,  and  then  reports  the  operational 
impact,  and  out  year  courses  of  action  (COAs)  based  on  funding  distribution  inputs. 
There  are  separate  performance,  schedule,  and  cost  modules.  A  thorough  review  of 
CPAT  has  been  performed  by  the  Naval  Postgraduate  School.^ 

CPAT  aids  in  decision  making  for  portfolio  resourcing  by  determining  the  "bang 
for  the  buck"  of  alternative  platform  mixes.  The  objective  is  to  maximize  fleet  per¬ 
formance  for  a  given  role  subject  to  constraints  for  budget,  schedule,  and  mini¬ 
mum/maximum  allowable  vehicle  numbers.  For  a  series  of  assumptions  on  vehi¬ 
cles,  roles,  availability,  etc.,  multiple  COAs  can  be  explored.  The  budget  constraint 
is  input  into  the  tool  (any  or  all  costs;  acquisition,  research,  development,  test,  and 
evaluation  (RDT&E),  O&S  and  the  performance  versus  time  is  determined  for  each 
funding  level  (the  COAs). 

The  tool  provides  fleet  composition  over  time  and  cost  breakdown  per  mission  per 
cost  metric  per  year.  There  are  3  parts  to  the  tool: 

1.  Performance  Module.  Value  of  the  fleet  mix  comprises  roles  that  must  be 
performed  by  vehicles,  and  each  role  is  assigned  a  mix  of  vehicles.  These 
roles  are  decomposed  into  major  attributes,  and  these  major  attributes  all  have 
measures.  Weightings  of  the  attributes  are  determined  using  a  swing  weight 
matrix,  and  weightings  of  the  roles  are  obtained  through  pair-wise  compari¬ 
son.  Single  utility  functions  (SUFs)  are  generated  from  subject  matter  expert 
input  for  each  measure  in  each  role.  Vehicle  performance  is  then  analyzed  for 
each  role,  and  these  performance  data  serve  as  input  into  the  SUFs  to  convert 
performance  into  utility.  Each  major  attribute  for  a  vehicle  in  a  role  is  then 
determined  by  summing  each  SUF,  which  is  a  function  of  performance,  mul¬ 
tiplied  by  the  attribute’s  weighting.  Each  vehicle  for  a  role  is  then  a  sum  of 
all  of  the  major  attribute  scores. 

2.  Schedule  Module.  Fleet  mixes  are  predefined,  as  are  availability  by  vehicle 
by  year. 

^Ewing  L,  Dell  RF,  MacCalman  M,  Whitney  L.  Capability  portfolio  analysis  tool  (CPAT)  ver¬ 
ification  and  validation  report.  Monterey  (CA):  Naval  Postgraduate  School;  2013  Jan.  Report  No.: 
NPS-OR-13-001. 
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3.  Cost  Module.  Cost  metrics  were  developed  to  represent  the  life  cycle  expec¬ 
tations,  and  attempted  to  capture  the  RDT&E,  procurement,  and  O&S  costs. 
These  are  not  life  cycle  cost  (LCC)  estimates,  but  rather  metrics  by  which  to 
rank  vehicle  alternatives. 

Classification  and  Regression  Trees  (CART)  (Salford  Systems)  is  a  commercial 
classification  tree  tool  used  for  data  mining  to  quickly  reveal  data  relationships. 
It  is  based  upon  the  mathematical  classification  theory  by  Jerome  Friedman  and 
Leo  Breiman.  Generally,  the  CART  method  maps  observations  about  an  outcome 
to  conclusions  about  the  outcome’s  target  value.  CART  models  can  be  translated 
into  SAS,  C,  Java,  and  Predictive  Model  Markup  Language  languages  for  imple¬ 
mentation  into  other  analysis  software.  Salford  Systems  offer  online  and  in-person 
training  and  support. 

ClearPoint  Strategy  (Ascendant  Strategy  Management  Group)  is  a  strategy 
management  tool  that  links  business  actions  and  decisions  to  business  strategy 
goals.  Projects,  measures,  and  objectives  can  all  be  linked  and  tracked.  The  tool  con¬ 
tains  organic  charting  capabilities  but  also  exports  to  the  Microsoft  Excel  spread¬ 
sheet  software.  Ascendant  offers  help  videos  and  documentation  free  on  their  web 
page,  and  also  offers  both  in-person  and  online  training. 

Comparion  Suite  (Expert  Choice,  Inc.)  is  a  web-based  tool  for  conducting  re¬ 
source  allocation  and  strategic  planning.  The  tool  appears  to  aid  in  portfolio  priori¬ 
tization.  The  company  website  provides  very  little  information  on  the  tool’s  capabil¬ 
ities.  Interested  parties  must  provide  contact  information  to  download  information. 

CyDesign  Studio  (CyDesign  Labs)  is  a  model-based  design  optimization  platform 
for  0-D  and  1-D  multiphysics  analysis.  Models  are  developed  in  the  open  source 
Modelica  language  and  compiled  within  the  CyDesign  tool.  CyDesign  Labs  was 
purchased  by  ESI  Group  in  October  2013.  CyDesign  Studio  integrates  requirements 
management,  design  space  exploration,  modeling,  Modelica  simulation,  tradespace 
studies,  parametric  optimization,  verification,  and  certification.  Users  can  view  their 
system  in  a  hierarchical  view  and  create  interfaces  between  components.  When  the 
models  are  simulated,  the  user  is  able  to  see  the  estimated  performance  capability 
achievable  by  the  system. 
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D3  (D3JS.org)  is  an  open-source  JavaScript  library  of  data  visualization  options, 
allowing  users  to  create  dynamic  and  interactive  visualizations.  D3  is  accessible 
through  Internet  browsers.  The  data  for  D3  can  be  in  JavaScript  Object  Notation  or 
comma-separated  value  format,  or  users  can  write  JavaScript  functions  to  read  in 
other  data  formats.  Essentially,  D3  is  a  library  of  visualization  overlays  for  data. 

DEA  SolverPro  (SAITECH,  Inc.)  is  based  upon  the  textbook  by  Cooper  et  al.^ 
Data  envelopment  analysis  is  a  nonparametric  method  to  estimate  production  fron¬ 
tiers.  DEA  SolverPro  uses  the  DEA  method  to  estimate  the  efficiency  of  production 
efficiency  of  an  entity.  DEA  does  not  have  an  input/output  function  and  instead 
assumes  each  entity  has  the  capability  to  produce  the  same  level  of  output  and  gen¬ 
erates  a  new  composite  frontier  from  the  best  producers.  The  textbook  referenced 
serves  as  the  main  training  tool  for  DEA  SolverPro. 

DecidelT  (Preference)  is  a  graphical  decision  analysis  tool  centered  on  decision 
trees  and  criteria  hierarchies.  The  tool  appears  to  follow  a  structured  decision  analysis 
approach,  with  users  defining  their  options,  identifying  decision  criteria,  assigning 
importance  weightings,  assigning  risk  levels,  and  conducting  sensitivity  analyses  on 
ranked  alternatives.  DecidelT  can  handle  multiple  objectives,  and  different  weight¬ 
ings  assigned  to  each  objective  by  different  stakeholders.  DecidelT  displays  rela¬ 
tionships  between  options,  objectives,  and  uncertainties  to  the  user.  Preference,  a 
Sweden-based  company,  offers  courses  and  seminars  to  train  users  in  the  use  of 
their  tools  as  well  as  in  decision  and  risk  analysis  theories. 

Decision  Explorer  (Banxia  Software  Ltd.)  is  an  idea  mapping  tool  that  enables 
the  user  to  visualize  the  reasoning,  structure,  and  interdependence  of  their  decision 
logic,  potential  outcomes,  and  risks.  The  focuses  are  idea  mapping,  scenario  build¬ 
ing,  brainstorming,  mind  mapping,  and  influence  diagramming.  Banxia,  a  British 
company,  offers  free  downloads  of  its  manuals  and  tutorials,  and  a  1-day  session  on 
how  to  use  the  tool. 

DecisionTools  Suite  6.0  (Palisade  Corporation)  is  a  set  of  programs  from  Pal¬ 
isade  Corporation  that  aide  in  decision  making.  The  suite  consists  of  ©RISK  for 
Monte  Carlo  simulations,  PrecisionTree  for  decision  trees,  TopRank  for  "what-if" 
analyses,  StatTools  for  statistical  analysis,  NeuralTools  for  neural  networks  and  pre- 

^Cooper  WW,  Seiford  LM,  Tone  K.  Data  Envelopment  Analysis:  a  comprehensive  text  with  mod¬ 
els,  applications,  references  and  DEA-solver  software.  New  York  (NY):  Springer  Science+Business 
Media,  EEC;  2007. 
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dictive  modeling,  and  Evolver  and  RISKOptimizer  for  optimization.  All  tools  are 
integrated  within  Microsoft  Excel  spreadsheet  software.  Palisade  Corporation  pro¬ 
vides  regional  training  events  as  well  as  live  web  training  and  onsite  training.  Tech 
support  and  consulting  services  are  also  available. 

Design  Analysis  toolKit  for  Optimization  and  Terascale  Applications  (DAKOTA) 
(Sandia  National  Laboratories)  is  an  open-source  toolkit  that  provides  an  exten¬ 
sible  interface  between  analysis  codes  and  a  variety  of  iterative  systems  analysis 
methods,  including  optimization,  uncertainty  quantification,  parameter  estimation, 
and  sensitivity  analysis.  DAKOTA  is  written  in  C-t-i-.  DAKOTA  operates  on  the  out¬ 
puts  of  a  user’s  analysis  codes  or  tools  using  built-in  algorithms.  Sandia  provides 
free  access  to  tutorials,  examples,  and  other  documentation.  DAKOTA  currently 
runs  in  a  Windows  DOS  command  prompt  environment,  but  Sandia  is  developing 
a  complementary  graphical  user  interface  (GUI).  DAKOTA  outputs  basic  statistics 
such  as  mean,  standard  deviations,  and  correlations.  Tabular  output  can  be  passed 
on  to  any  third  party  statistical  analysis  tool. 

D-Sight  Desktop  (D-Sight  Inc.)  is  multicriteria  decision-making  software  that  im¬ 
plements  the  Preference  Ranking  Organization  Method  for  Enrichment  Evaluation 
and  Geometrical  Analysis  for  Interactive  decision  Aid  methods.  The  preference 
modeling  process  scores  alternatives  by  comparison  in  pairs  and  generating  a  pref¬ 
erence  function  that  are  used  to  assess  the  best  alternative.  D-Sight  includes  plug¬ 
ins  to  elicit  user  weightings  for  each  criteria  through  an  interactive  process.  D-Sight 
has  capabilities  to  export  reports  to  Microsoft  Word  and  Excel  office  productivity 
software,  as  well  as  Adobe  Systems  Portable  Document  Eormat.  It  can  conduct  sen¬ 
sitivity  analyses  and  produce  data  visualizations.  D-Sight  Inc.  provides  case  studies, 
user  examples,  online  tutorials,  and  documentation  for  training.  D-Sight  also  offers 
D-Sight  Web,  a  collaborative  version  of  the  D-Sight  Desktop  multicriteria  decision¬ 
making  software.  D-Sight  Web  is  accessible  online  and  allows  for  collaboration 
from  multiple  users  on  the  same  project. 

Enterprise  Portfolio  Simulator  (ProModel  Corporation)  is  a  project  portfolio 
planning,  simulation,  and  managing  tool  based  on  the  ProModel  Corporation’s  Port¬ 
folio  Simulator  tool  that  provides  a  web  browser  based  interface  to  allow  for  collab¬ 
oration  on  multiple  projects  by  multiple  users.  The  discrete  event  simulations  can  be 
analyzed  and  optimized  to  improve  project  performance.  Enterprise  Portfolio  Sim¬ 
ulator  can  be  connected  with  Microsoft  Project  project  and  portfolio  management 
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software  so  that  what  if  scenarios  can  be  run  and  analyzed.  ProModel  Corporation 
provides  online  and  classroom  based  training  programs  and  provides  online  sup¬ 
port. 

Equity3  (Catalyze  Ltd.)  is  a  multicriteria  decision  analysis  tool.  An  individual 
user  constructs  a  decision  model  by  assessing  the  cost  and  benefits  of  multiple  cri¬ 
teria  and  has  a  graphical  tool  to  elicit  accurate  weights  for  each.  EquityS  allows  for 
rapid  what  if  analyses  based  upon  given  resources  and  provides  a  frontier  of  avail¬ 
able  options.  EquityS  is  geared  toward  budgeting  and  resource  allocation  decisions. 
Catalyze  Etd.  provides  12-month  support  contracts  that  include  unlimited  technical 
support  over  telephone,  email,  and  fax  in  addition  to  optional  onsite  assistance. 

Eureqa  Desktop  (Nutonian,  Inc.)  was  originally  developed  by  Cornell  University, 
but  has  transitioned  to  ownership  by  a  private  company.  Eureqa  uses  symbolic  re¬ 
gression  to  find  linear  and  nonlinear  mathematical  relationships  in  data  provided 
as  rows  and  columns.  The  user  can  specify  what  operators  to  use  (e.g.,  math,  trig, 
logic,  etc.)  in  the  target  expression,  and  Eureqa  identifies  the  "simplest"  mathe¬ 
matical  relationship.  Simplicity  of  the  relationship  is  determined  from  the  number 
of  operators  (e.g.,  addition,  subtraction,  power,  etc.).  Eureqa  displays  all  identified 
equations.  Eureqa  for  Excel  started  out  as  a  free  download  and  use  tool,  but  recently 
there  is  a  new  Eureqa  Desktop  product  that  is  installation-only  and  available  for  a 
free  30-day  trial. 

Excel  (Microsoft)  is  a  widely  used  data  analysis  and  visualization  tool.  Users  enter 
data  in  a  row-column  format  and  use  built-in  menus  to  operate  on  the  data.  With 
the  Visual  Basic  for  Applications  (VBA)  programming  language,  users  can  expand 
the  capabilities  of  Excel  to  conduct  any  of  the  identified  tradespace  exploration 
functions.  Many  other  tradespace  exploration  tools  can  export  to  or  import  from 
Excel-based  files  given  its  wide  user  base.  Many  vendors  offer  training,  and  there 
is  a  very  large  user  community  who  share  tips  and  tricks  online. 

ExtendSim  Suite  (Imagine  That,  Inc.)  is  a  dynamic  process  simulation  tool  that 
allows  users  to  predict  the  impact  that  changes  will  have  on  both  new  and  existing 
processes.  Users  construct  their  models  with  the  built-in  "building  blocks"  and  can 
execute  their  continuous  or  discrete  processes  deterministically  or  probabilistically. 
The  tool  provides  visual  feedback  of  the  process  execution  as  well  as  data  analytics 
and  charting  capabilities.  ExtendSim  offers  training  videos  on  their  web  page,  both 


35 


for  their  specific  products  and  for  general  themes  and  functions  that  their  products 
cover.  There  is  a  broken  link  to  a  user  forum  on  the  web  page.  ExtendSim  has  a  net¬ 
work  of  "Expert  Solutions  Providers"  that  work  with  customers  to  apply  ExtendSim 
products  to  their  specific  needs. 

ForeTell  (Decision  Path,  Inc.)  is  a  GUTdriven  project  and  portfolio  management 
tool  that  allows  users  to  build  scenarios,  execute  the  scenarios  probabilistically, 
and  then  conduct  statistical  analysis  on  the  results.  Users  establish  the  key  perfor¬ 
mance  indicators  to  track  during  simulation  such  that  the  results  are  displayed  as 
likelihood  of  outcome  for  each  indicator,  per  decision,  per  scenario.  The  top-level 
scenario  planning  approach  is  applicable  to  market  penetration,  disaster  planning, 
and  portfolio  management.  Help  is  available  by  contacting  the  company  via  email 
or  phone. 

Framework  for  Assessing  Cost  and  Technology  (FACT)  (Georgia  Tech  Re¬ 
search  Institute  [GTRI])  is  a  government-owned,  open  architecture,  browser-based 
framework  for  assessing  performance  and  cost  of  mechanical  systems,  funded  by 
the  US  Navy  through  the  Marine  Corps  Systems  Command.  FACT  uses  systems 
modeling  language  (SysME)  in  an  MBSE  environment.  FACT  enables  "near  real¬ 
time  analysis  for  exploring  the  design  parameter  trades  that  affect  the  overall  per¬ 
formance,  reliability,  and  cost  of  a  system  design. FACT  is  hosted  on  a  US 
Navy  server  and  requires  common  access  card  (CAC)  credentials  to  log  in.  Users 
map  a  system’s  work  breakdown  structure  (WBS),  in  MIE-STD-881A  format,  to 
SysME  block  definition  diagrams  to  represent  the  decomposed  physical  hierarchy 
in  SysME.  Performance  relationships  are  captured  in  SysME  parametric  diagrams, 
and  can  be  calculated  using  user-provided  physics-based  equations  or  tabular  data. 
The  user  builds  their  system  in  a  bottom-up  fashion,  by  selecting  components  from 
a  component  library.  Components  must  already  be  characterized  with  value  prop¬ 
erties  (e.g.,  torque,  power,  mass,  speed,  etc.)  by  the  user.  These  values  are  used  as 
inputs  to  the  parametric  block  diagram,  which  then  links  out  to  surrogates  or  mod¬ 
eling  and  simulation  (M&S)  tools.  The  outputs  from  the  surrogate  equations  and 
M&S  tools  are  then  returned  to  FACT  through  the  parametric  diagram. 

"^Ender  TR,  Browne  CD,  Yates  WW,  O’Neal  M.  FACT;  an  M(&S  framework  for  systems  engi¬ 
neering.  Paper  presented  at:  Interservice/Industry  Training,  Simulation,  and  Education  Conference 
(1/lTSEC).  2012  Dec  3-6;  Orlando,  EL. 

^Ender  TR.  Model-based  systems  engineering  as  a  collaborative  web-service.  Paper  presented 
at:  National  Defense  Industrial  Association  Ground  Vehicle  Systems  Engineering  and  Technology 
Symposium  (NDIA  GVSETS).  2013  Aug  20-22;  Troy,  MI. 
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FACT  contains  a  low  level  of  computer- automated  design  compatibility;  users  can 
drag-and-drop  an  image  file  into  the  tool  to  "see"  the  part  they  are  updating/mod¬ 
ifying.  Image  files  are  linked  to  sections  of  the  WBS  so  that  the  user  can  click  on 
a  part  in  the  image  and  jump  to  that  section  of  the  WBS  for  editing  values.  FACT 
can  instantiate  many  (thousands)  system  models  using  an  internal  algorithm  that 
generates  a  WBS  at  random.  Each  WBS  is  then  analyzed  for  its  capabilities  using 
user-provided  data  and  models. 

For  tradespace  exploration,  users  set  threshold  and  objective  values  on  key  perfor¬ 
mance  parameters,  and  with  user- specified  ranges  and  distributions  on  input  vari¬ 
ables  FACT  will  produce  PDF  and  CDF  plots  depicting  the  likelihood  of  achieving 
values  of  interest.  FACT  has  organic  plotting  capability  including  scatterplots  that 
allow  highlighting  of  groups  of  points  or  filtering  based  on  minimum/maximum 
plot  axis  values,  as  well  as  small  multiples  plots  that  allow  comparison  of  outputs 
with  each  other. 

Genetic  Resources  for  Innovation  and  Problem  Solving  (GRIPS)  (Aerospace 
Corporation)  is  a  multiobjective  decision  support  tool  developed  by  the  Aerospace 
Corporation  and  the  Penn  State  Applied  Research  Laboratory.  Development  started 
as  a  single-objective  optimization  tool  for  National  Security  Space.  The  tool  was 
later  expanded  to  multiobjective  optimization.  GRIPS  uses  evolutionary  algorithms 
for  exploration  (search)  and  exploitation  (selection)  of  the  most  fit  solutions;  the 
nondominated  set  of  solutions  found  on  the  Pareto  frontier.  There  are  3  stages  to 
the  GRIPS  decision  support  process: 

1.  Interviews 

2.  Multiobjective  optimization 

3.  Visualization  and  decision  making 

(a)  Identify  and  understand  design  parameters,  objectives,  and  constraints 

(b)  Stakeholder  models  or  other  models  are  integrated  into  GRIPS 

(c)  Pareto  set  is  exported  to  the  Aero  Vis  tool  for  visualization  through  plot¬ 
ting  in  up  to  7  dimensions 
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Internet  searches  for  GRIPS  result  in  patents,  but  there  is  scarce  technical  detail 
available  via  publications  and  case  studies.  Little  information  is  available  publicly 
after  2010. 


GoldSim  Pro  (GoldSim  Technology  Group)  is  an  add-in  to  the  Microsoft  Excel 
spreadsheet  software  that  allows  users  to  set  ranges  and  distributions  on  inputs  of 
Excel-based  models.  The  product  operates  similarly  to  Oracle’s  Crystal  Ball  and 
Erontline  System’s  Risk  Solver  Pro  add-ins.  After  users  establish  the  input  distri¬ 
butions  and  the  number  of  trials,  histograms  and  simulation  statistics  are  visible  for 
review  within  Microsoft  Excel.  GoldSim  Pro  can  conduct  discrete  and  continuous 
event  simulations  after  building  systems  with  a  hierarchical  approach.  GoldSim  Pro 
is  better  suited  for  complex  systems  with  loose  interactions  than  it  is  for  engineered 
systems  with  well-defined  interactions. 

gPROMS  (Process  Systems  Enterprise  Limited)  is  a  model-based  engineering 
approach  for  high  fidelity  predictive  modeling  in  the  process  industry.  gPROMS  is 
a  family  of  modeling  software,  with  specific  modules  and  libraries  for  multiphase 
fluid  flow,  fuel  cells,  depressurization,  crystallization,  carbon  capture  and  storage, 
and  molecular  thermodynamics.  Includes  a  wrapper  that  wraps  gPROMS  models 
and  solvers  such  that  they  can  be  implemented  in  stand-alone  analysis  environments 
such  as  MATEAB.  Appears  to  have  the  capability  to  develop  custom  models  using 
the  ModelBuilder  tool,  which  allows  drag-and-drop  of  hardware  into  a  model  can¬ 
vas.  Erom  there,  users  can  simulate  the  model,  including  uncertainty  on  inputs.  Data 
analysis  and  visualization  capabilities  are  unknown.  Process  Systems  Enterprise  of¬ 
fers  in-person  training  as  well  as  online  support  via  a  hypertext  markup  language 
form. 

Hiview3  (Catalyze  Ltd.)  is  a  decision  modeling  tool  that  is  effective  as  a  collabo¬ 
rative  tool  supporting  decision  conferences  and  the  evaluation  of  decision  options. 
The  modeling  process  is  similar  to  EquityS  in  that  the  decision  tree  is  constructed 
followed  by  scoring  criteria  and  analyzing  the  model  to  determine  the  recommended 
set  of  available  options.  Catalyze  Etd.  provides  12-month  support  contracts  that 
include  unlimited  technical  support  over  telephone,  email,  and  fax  in  addition  to 
optional  onsite  assistance. 

Iris  (Ayasdi)  is  a  commercial  "insight  discovery"  data  analysis  software.  It  is  a 
model-free  and  code-free  data  mining  tool  that  runs  through  hundreds  of  algo- 
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rithms  to  identify  relationships  within  data  sets  by  using  their  unique  topological 
data  analysis  approach.  It  has  a  cloud-based  option  to  offload  computations  from 
the  local  user’s  machine.  Its  GUI  provides  visualization  that  graphically  depicts 
links  between  data  points  in  multiple  dimensions.  Its  use  in  the  medical  industry 
found  links  between  breast  cancer  survivors  and  their  genetic  makeup.  Iris  is  purely 
a  data  analysis  tool  and  does  not  provide  any  modeling  or  simulation  capabilities. 
Ayasdi  provides  online  product  support. 

Isight  (Dassault  Systemes)  is  a  commercial  visual  multidisciplinary  analysis  and 
optimization  tool  that  is  part  of  Dassault  Systemes’  SIMULIA  family  of  software. 
Isight  provides  a  drag  and  drop  workflow  environment  to  integrate  simulation  mod¬ 
els  from  multiple  disciplines  and  software  and  automates  the  execution  of  simu¬ 
lations.  The  resulting  data  can  be  post  processed  with  Isight  to  explore  the  design 
space  and  visualize  the  data.  Optimization  and  design  of  experiments  (DOE)  can  be 
applied  to  the  workflow  to  automate  the  design  and  analysis  of  an  optimal  or  best 
solution.  Isight  includes  components  that  can  interface  with  Microsoft  Excel  and 
Word  office  productivity  software,  MATEAB,  Python  scripts,  SIMUEIA  software, 
and  other  third  party  analysis  tools. 

JMP  (SAS)  is  a  commercial  statistical  discovery  program  from  SAS.  JMP  is  very 
graphically  driven  and  leans  heavily  towards  interactive  data  visualization  and  sta¬ 
tistical  simulation.  It  does  not  focus  on  optimization.  Exploratory  statistical  analysis 
can  be  used  to  identify  trends  and  various  plotting  tools  can  graphically  depict  them. 
Models  can  be  created  through  various  probabilistic  methods  and  can  be  analyzed 
through  simulations  and  more  plotting  tools.  JMP  has  scripting  functionality  for 
automating  analyses  and  recreating  results.  JMP’s  analysis  capabilities  are  selected 
from  built  in  functions,  no  coding  or  scripting  is  necessary.  It  can  interface  with 
other  software  such  as  SAS,  Microsoft  Excel  spreadsheet  software,  and  R  language 
programs.  SAS  offers  online  resources,  community  support  discussions,  live  web 
training,  and  in-person  training  courses. 

Joint  Integrated  Analysis  Tool  (JIAT)  (Assistant  Secretary  of  the  Army  for 
Financial  Management  and  Comptroller)  is  a  CAC-enabled,  web-based  system 
that  provides  a  single  log-on/user  interface  where  analysts  can  search  for,  retrieve, 
normalize,  and  incorporate  official  validated  data  into  their  cost  estimates  and  mod¬ 
els.  JIAT  is  government-only  access  and  employs  strategies  to  manage  and  con¬ 
trol  access  to  sensitive  information  under  3  main  headings:  databases,  libraries. 
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and  models.  JIAT  provides  access  to  8  major  database  sources,  over  725  cost  es¬ 
timating  relationships  and  factors,  over  125  web  links,  and  models  from  4  major 
tool  providers.  JIAT’s  databases  include  ACDB,  AMCOS,  CKB,  DAMIR,  DCARC 
CSDR-SR,  FLIS,  FORCES,  Government  Rates,  and  OSMIS. 

Users  initiate  a  JIAT  session,  which  is  an  instance  of  querying  a  database  or  running 
a  model.  Next,  the  user  selects  from  a  list  of  providers,  which  are  the  data  and 
model  sources,  and  JIAT  displays  the  available  data  and  models.  The  user  will  have 
to  understand  the  inputs  and  outputs  of  the  models,  and  have  access  to  that  data. 
Once  a  cost  estimate  is  generated,  JIAT  results  can  be  exported  to  Microsoft  Excel 
spreadsheet  software,  CO$TAT,  and  ACE  for  further  analysis. 

JIAT  can  perform  non-time  phased  or  time-phased  analysis  to  perform  what-if  and 
sensitivity  analyses.  Outputs  are  displayed  in  organic  plotting  services,  or  tabular 
data  can  be  exported  for  analysis  by  a  third  party  tool.  Eree  training  is  offered  pe¬ 
riodically.  The  JIAT  web  pages  provide  access  to  a  user  manual,  knowledge  base 
articles,  newsletters,  and  brown  bag  presentations. 

KNITRO  (Ziena  Optimization  LLC)  is  a  nonlinear  optimization  solver  imple¬ 
menting  the  Interior-point  Direct  algorithm,  the  Interior-point  CG  algorithm,  and 
the  Active  Set  algorithm.  KNITRO  is  available  with  interfaces  to  multiple  pro¬ 
gramming  languages  and  computational  tools  such  as  C-t-i-,  Python,  MATEAB,  Mi¬ 
crosoft  Excel,  and  Eabview.  Ziena  provides  support  contracts  that  include  software 
maintenance,  usage  support,  and  consultation  (at  additional  cost). 

Logical  Decisions  for  Windows  (Logical  Decisions)  is  a  decision  analysis  tool.  It 
provides  7  methods  for  a  user  to  apply  weights  to  different  system  metrics,  orga¬ 
nizes  metrics  into  a  hierarchical  chart,  analyzes,  and  scores  a  system.  No  optimiza¬ 
tion  or  system  selection  is  done  with  this  tool.  The  results  from  a  Logical  Decisions 
for  Windows  analysis  can  be  used  to  optimize  about  outside  of  this  tool.  Logical 
Decisions  offers  unlimited  over  the  phone  and  email  technical  support  as  well  as  in 
person  seminars  and  consulting  services. 

Logical  Decisions  Portfolio  (Logical  Decisions)  is  based  on  Logical  Decisions 
for  Windows.  It  is  a  decision  analysis  tool  that  analyzes  a  set  of  alternatives  and 
can  aid  in  selecting  a  set  of  solutions  rather  than  a  single  solution.  Cost  and  other 
metrics  can  be  applied  to  portfolio  alternatives  and,  combined  with  the  rankings  of 
each  alternative  from  Logical  Decisions  for  Windows,  Logical  Decisions  Portfolio 
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applies  an  optimization  algorithm  to  find  the  best  solution.  Logical  Decisions  offers 
unlimited  over  the  phone  and  email  technical  support  as  well  as  in-person  seminars 
and  consulting  services. 

MapleSim  (Maplesoft)  is  a  commercial  modeling  and  simulation  software  from 
Maplesoft.  It  uses  an  object-oriented  style  of  model  building  by  dragging  and  drop¬ 
ping  components  into  the  model.  MapleSim  automatically  generates  the  model 
equations  and  can  then  run  simulations  and  analyses.  MapleSim  uses  the  mathemat¬ 
ical  engine  of  Maple,  which  is  another  piece  of  commercial  computational  software 
developed  by  Maplesoft.  The  components  in  MapleSim  are  built  from  the  Model- 
ica  open  source  modeling  language.  Maplesoft  provides  training  videos,  example 
models,  online  documentation  and  training  courses. 

Mathematica  (Wolfram)  is  a  commercial  computational  environment.  It  is  simi¬ 
lar  to  MATLAB  but  it  uses  a  more  visually  interactive  front  end  and  has  several 
features  to  produce  stand-alone  mathematical  applications  such  as  interactive  data 
visualizations,  interactive  text  books,  and  website  applets.  In  the  GUI,  computa¬ 
tions  are  displayed  as  notebook  style  documents.  Mathematica  has  built-in  func¬ 
tions  for  symbolic  computation,  optimization,  data  visualization,  signal  process¬ 
ing,  statistical  analysis,  automated  report  generation,  and  more.  Mathematica  can 
link  to  other  mathematical  software  packages  including  Microsoft  Excel,  MAT- 
LAB,  and  R.  Other  programming  languages  such  as  VBA  and  Python  can  connect 
to  Mathematica.  It  is  widely  used  in  engineering,  finance,  analytics,  and  software 
development,  and  as  a  learning  tool  for  academic  math  and  sciences.  Wolfram  pro¬ 
vides  options  for  learning  Mathematica  online,  in  person,  or  by  documentation. 

MATLAB  (Mathworks)  is  a  commercial  numerical  computing  environment  and 
programming  language  from  Mathworks.  It  can  be  expanded  by  add-on  toolboxes 
ranging  from  statistical  analysis  to  control  systems  design  and  more.  Included  with 
MATLAB  is  Simulink,  a  graphical  block  diagramming  tool  used  for  modeling,  sim¬ 
ulation,  and  analysis  of  dynamical  systems.  Simulink  is  typically  used  for  signal 
processing  and  control  systems  design  and  testing.  MATLAB  is  widely  used  in  in¬ 
dustry  and  academia.  Users  can  create  custom  programs  in  a  similar  fashion  as  a 
standard  programming  language  such  as  Python  or  C-i-i-.  Programs  can  be  exported 
to  run  without  requiring  a  local  installation  of  MATLAB  software.  Mathworks  pro¬ 
vides  online  support  and  training  and  many  third  party  books  and  training  courses 
are  commonly  available.  MATLAB  is  similar  in  use  to  Maple  from  Maplesoft. 
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MedModel  Optimization  Suite  (ProModel  Corporation)  is  a  planning  and  through¬ 
put  analysis  tool  for  the  healthcare  industry.  The  tool  can  analyze,  visualize,  and 
optimize  the  layout  of  a  medical  facility  based  on  performance  metrics  specified 
by  the  user.  Users  can  also  pose  what-if  questions  about  the  layout  and  its  con¬ 
straints,  and  the  tool  will  provide  options  for  improving  the  performance  metrics. 
The  tool  has  import/export  capabilities  with  Microsoft  Excel  spreadsheet  software. 
ProModel  Corporation  provides  online  and  classroom  based  training  programs  and 
provides  online  support. 

modeFRONTIER  (Esteco)  is  a  commercial  software  for  multiobjective  optimiza¬ 
tion  and  multidisciplinary  optimization.  It  integrates  with  third  party  analysis  soft¬ 
ware  (e.g..  Computational  Fluid  Dynamics  [CFD],  Computational  Structural  Dy¬ 
namics,  etc.)  to  create  an  environment  by  which  the  design  process  can  be  auto¬ 
mated  to  optimize  a  design  and  provide  data  analysis  and  visualizations  for  de¬ 
sign  space  exploration  to  aid  in  decision  making.  modeFRONTIER  is  comparable 
to  ModelCenter  in  that  it  is  an  environment  that  links  various  analysis  tools  to¬ 
gether  to  streamline  and  automate  the  design  process.  The  modeFRONTIER  envi¬ 
ronment  sets  up  the  design  process  for  rapid  design  modification  and  re-simulation 
and  analysis.  modeFRONTIER  is  in  use  by  many  internationally  recognized  uni¬ 
versities  and  corporations. 

ModelCenter  (Phoenix  Integration)  is  a  commercial  graphical  environment  for 
systems  engineering  and  optimization  aimed  at  reducing  life  cycle  cost  and  time. 
Several  aspects  of  the  engineering  process  (e.g.,  CFD  and  finite  element  analysis 
[FEA]  analyses)  can  be  linked  together  with  ModelCenter  to  generate  a  design 
space  that  can  be  analyzed,  explored,  optimized  and  visualized.  Probabilistic  meth¬ 
ods  can  be  utilized  to  model  high  fidelity  problems  and  then  simulated  to  reduce 
time.  Designs  can  be  optimized  through  the  use  of  30  algorithms  from  leading  re¬ 
search  organizations.  Interactive  visualizations  are  also  possible.  ModelCenter  cap¬ 
tures  the  global  system  engineering  process  in  one  package.  Phoenix  Integration 
provides  webinars,  online  support,  training  courses,  and  consultations. 

Multivariate  Adaptive  Regression  Splines  (MARS)  (Salford  Systems)  is  a  com¬ 
mercial  nonlinear  regression  modeling  tool.  Its  strength  is  the  capability  to  capture 
nonlinear  relationships  using  the  MARS  regression  method.  Regression  models  can 
be  used  for  data  mining  applications.  Salford  Systems  offers  online  and  in-person 
training  and  support. 
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Nexus  (iChrome  Ltd.)  is  a  commercial  systems  design  optimization  software  sim¬ 
ilar  in  role  to  ModelCenter  and  modeFRONTIER.  A  GUI  links  each  analysis  in  the 
design  process.  Third  party  tools  such  as  Nastran,  Abaqus,  Fluent,  ANSYS,  and 
MATLAB  can  be  linked  to  streamline  and  automate  the  design  and  analysis  pro¬ 
cess.  The  flowchart  module  is  a  graphical  representation  of  the  design  process  and 
is  where  evaluation  nodes  can  be  linked  to  external  software.  The  response  surfaces 
module  can  create  static  and  dynamic  DOE  and  can  create  response  surfaces.  The 
visualization  and  post-processing  module  can  create  2-D  and  3-D  charts  to  display 
results.  Nexus  is  available  for  Windows  and  Einux  and  in  local  or  scalable  versions 
with  free  licenses  available  for  academia  and  research  purposes. 

OpenMDAO  (National  Aeronautics  and  Space  Administration  (NASA)  Glenn 
Research  Center)  is  an  open  source  Python  based  Multidisciplinary  Design  Analysis 
and  Optimization  (MDAO)  tool.  OpenMDAO  has  built-in  functions  for  construct¬ 
ing  and  executing  optimization  and  statistical  analyses.  Third  party  analysis  tools 
can  be  integrated  into  an  analysis  model  through  built-in  plug-ins  or  from  user- 
created  wrappers.  OpenMDAO  can  provide  multiple  levels  of  fidelity  through  the 
number  and  fidelity  of  the  analysis  tools  utilized  by  the  user.  The  automation  of 
analyses  and  optimization  provides  a  streamlined  method  for  design  and  analysis 
of  systems.  Python  packages  required  for  OpenMDAO  also  give  numerical  compu¬ 
tation  and  data  visualization  capabilities.  Given  a  user’s  level  of  coding  expertise, 
OpenMDAO  can  function  in  a  similar  capacity  to  other  MDAO  tools  such  as  Mod¬ 
elCenter.  NASA  Glenn  is  still  actively  developing  and  updating  OpenMDAO  and 
provides  support  tutorials,  videos,  and  online  discussion  forums. 

OptiY  (OptiY  GmbH)  is  a  commercial  software  for  multidisciplinary  analysis  and 
optimization.  Similar  to  ModelCenter  and  OpenMDAO,  OptiY  has  optimization 
algorithms  and  integrates  third  party  analysis  tools  (e.g.,  MATEAB,  ANSYS)  to 
create  a  design  environment  to  streamline  the  design  process  and  aid  in  decision 
making.  OptiY  has  a  graphical  flowchart  style  workflow  editor  where  users  can 
integrate  external  tools.  It  has  data  mining  capabilities  to  identify  trends  and  rela¬ 
tionships  within  data  sets.  It  can  conduct  local  and  global  sensitivity  studies  and 
probabilistic  simulations  to  characterize  realistic  system  behaviors.  OptiY  can  cre¬ 
ate  metamodels  or  surrogate  models  to  conduct  virtual  optimizations  that  can  speed 
up  the  design  process.  Metamodels  can  be  exported  to  C,  Modelica,  and  even  MAT¬ 
EAB  codes  for  further  simulations  outside  of  OptiY.  OptiY  provides  online  tutorial 
videos  and  scripts  for  integrating  several  popular  commercial  analysis  tools. 
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Portfolio  Simulator  (ProModel  Corporation)  is  a  strategic  and  tactical  portfolio 
management  tool.  Users  can  operate  the  tool  top-down  by  specifying  their  strate¬ 
gic  corporate  objectives,  such  as  net  present  value  and  return  on  investment,  and 
allow  the  tool  to  optimize  on  tactical  and  operational  goals,  such  as  facility  metrics, 
resource  loading,  schedule,  and  more.  Or  the  tool  can  be  run  in  reverse  (bottom- 
up).  The  tool  integrates  with  Microsoft  Project  and  Excel  software.  Decision  trees, 
charts,  and  tables  allow  users  to  make  decisions  based  on  the  outcomes.  ProModel 
Corporation  provides  online  and  classroom  based  training  programs  and  provides 
online  support. 

Process  Simulator  (ProModel  Corporation)  is  a  Lean  Six  Sigma  and  value  stream 
focused  tool.  The  tool  is  a  plug-in  to  the  Microsoft  Visio  flowcharting  software  to 
help  manufacturers  design  and  re-engineer  their  business  and  manufacturing  pro¬ 
cesses.  Users  can  create  value  stream  maps,  estimate  product  workflow  outputs  and 
times,  and  estimate  resource  loading  required  for  meeting  process  metrics.  The  tool 
provides  top-down  views  of  processes  such  that  the  user  can  visualize  movements 
and  limitations  of  processes.  The  objective  is  to  capture  interdependencies  through 
process  simulation  and  optimization.  ProModel  Corporation  provides  online  and 
classroom  based  training  programs  and  provides  online  support. 

Project  Simulator  (ProModel  Corporation)  is  a  scenario  planning,  simulation, 
and  reporting  add-in  for  the  Microsoft  Project  software.  Project  execution  interde¬ 
pendency  and  variability  can  be  simulated  and  then  plotted.  Distributions  can  be  set 
on  project  inputs  and  schedule  impacts  can  be  seen  directly  in  the  Windows-based 
project  and  portfolio  management  software.  ProModel  Corporation  provides  online 
and  classroom  based  training  programs  and  provides  online  support. 

Promax  Professional  (Cogentus  Consulting  Ltd)  is  a  project  and  portfolio  man¬ 
agement  tool  for  determining  combinations  of  projects  or  activities  that  meet  bud¬ 
get  constraints.  Applications  include  prioritization  and  evaluation  for  choosing  be¬ 
tween  projects,  technologies,  software,  partners,  or  strategies.  Promax  contains  a 
root  cause  analysis  capability  as  well  as  an  innovation  and  idea  generation  tool. 
The  tool  contains  organic  drawing  and  charting  capabilities.  Cogentus  offers  online 
video  tutorials,  within-product  help  menus,  downloadable  manuals,  and  other  help 
documentation  on  their  web  page. 
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ProModel  Optimization  Suite  (ProModel  Corporation)  is  a  discrete  event  sim¬ 
ulation  tool.  The  tool  can  analyze,  visualize,  and  optimize  on  multiple  scenarios 
related  to  supply  chain,  logistics,  manufacturing,  and  operational  scenarios  based 
on  performance  metrics  specified  by  the  user.  Users  can  also  pose  what-if  questions 
about  event-based  occurrences  of  a  process,  and  the  tool  will  provide  options  for 
improving  the  performance  metrics.  The  tool  has  import/export  capabilities  with 
Microsoft  Excel  spreadsheet  software.  ProModel  Corporation  provides  online  and 
classroom  based  training  programs  and  provides  online  support. 

R  (R  Foundation)  is  an  open  source  programming  language  and  environment  for 
statistical  computing  and  graphics.  R  by  itself  uses  a  command  line  interface  but 
third  party  GUIs  are  available.  R  is  extensible  through  installation  of  packages  from 
a  growing  library  supported  by  the  open  source  community.  R  is  based  off  of  S,  a 
commercially  developed  statistical  programming  language  from  Bell  Laboratories. 
Most  codes  written  in  S  will  run  through  R  without  modification.  R’s  strength  is 
that  it  can  produce  high  quality  graphics  for  data  visualization  with  relative  ease. 
It  can  also  be  linked  to  C-t-i-  or  Fortran  code  for  computationally  heavy  programs. 
R  is  widely  accepted  and  used  for  statistical  analysis  and  visualization  application 
in  many  industries.  Documentation  and  learning  tools  are  widely  available  in  print 
and  online  without  cost. 

RandomForests  (Salford  Systems)  is  a  commercial  data  mining  tool  that  builds 
off  of  Salford’s  CART  methodology.  Its  strength  is  the  capability  to  generate  and 
assess  multiple  decision  trees  such  that  variable  importance  can  be  ranked.  The  user 
must  have  familiarity  with  the  structure  of  Random  Forests,  including  the  shape, 
predictors,  and  how  to  implement  randomness  in  the  generation  of  leaves.  Salford 
Systems  offers  online  and  in  person  training  and  support. 

Rank  Inclusion  in  Criteria  Hierarchies  (RICH)  Decisions  (Systems  Analysis 
Laboratory  [SAL],  Aalto  University)  is  a  web-based,  Java  decision  support  soft¬ 
ware  based  on  the  RICH  method  that  is  used  to  capture  and  analyze  preferences. 
In  the  RICH  method,  the  decision  maker  can  specify  subsets  of  attributes  that  con¬ 
tain  the  most  important  attribute  or  associate  a  set  of  rankings  with  a  given  set  of 
attributes.  It  assigns  alternatives  a  ranking  in  each  of  the  desired  decision  attributes 
by  asking  the  user  to  provide  answers  to.  How  well  does  alternative  perform  with 
respect  to  one  attribute?  and  How  well  does  one  alternative  perform  with  respect 
to  all  attributes?  The  tool  allows  users  to  provide  lower  and  upper  limits  of  alterna- 
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tive  performance,  as  well  as  weight  the  attributes.  The  University  provides  limited 
help  and  limited  documentation  on  their  web  pages.  Their  web  pages  are  confusing, 
have  poor  documentation,  and  it  is  unclear  how  it  fits  into  a  larger  suite  of  capabil¬ 
ities.  RICH  is  a  part  of  the  Decisionarium  site  for  interactive  multicriteria  decision 
support  tools. 

Rave  (Aerospace  Systems  Design  Laboratory  [ASDL],  Georgia  Institute  of 
Technology)  is  an  open  source  decision  support  tool  with  capabilities  of  visual¬ 
ization,  optimization,  DOE,  decision  analysis,  and  surrogate  modeling.  Rave  is  a 
MATLAB  toolbox  and  can  interface  with  other  computational  scripts  written  in 
MATLAB  because  it  is  built  upon  MATLAB  m-code.  Rave  allows  a  user  to  perform 
continuous  or  discrete  visualization,  letting  the  decision  maker  view  the  continuous 
impact  of  independent  variables  on  dependent  variables,  as  well  as  seeing  the  same 
design  points  in  multiple,  discrete  views.  Rave  can  generate  surrogate  models  (arti¬ 
ficial  neural  networks  and  response  surface  equations)  of  a  data  set  for  exploration 
of  the  design  space.  Rave  cannot  link  multiple  codes  together.  Rave  can  perform  op¬ 
timization  through  generation  of  new  designs  to  meet  objective  functions,  as  well 
as  through  sorting  existing  designs.  Rave  has  capabilities  similar  to  ATSV,  IMP, 
ModelCenter,  and  OpenMDAO. 

Relational- Oriented  Systems  Engineering  and  Technology  Tradeoff  Analysis 
(ROSETTA)  (ASDL,  Georgia  Institute  of  Technology)  is  a  methodology  to  "bridge 
the  gap  between  qualitative  subject  matter  expert  driven  techniques  and  quantita¬ 
tive  modeling  and  simulation  techniques."  This  methodology  utilizes  quality  func¬ 
tion  deployment  (QFD),  which  is  mapped  to  modeling  and  simulation  processes. 
Surrogate  models  created  through  response  surface  equation  (RSE)  methodology 
model  the  relationships  from  QFD  and  also  allow  for  Monte  Carlo  simulation  of 
requirements.  The  ROSETTA  methodology  is  targeted  for  use  in  the  early  stages 
of  systems  design  to  help  quantitatively  define  design  space  boundaries  and  subject 
matter  expert  decisions. 

RiskSim  (TreePlan  Software)  is  a  Monte  Carlo  simulation  add-in  for  the  Microsoft 
Excel  spreadsheet  software.  RiskSim  provides  random  number  generator  functions 
as  inputs  for  Excel-based  models  and  conducts  Monte  Carlo  simulation.  TreePlan 
provides  answers  to  frequently  asked  questions  for  their  software. 
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Risk  Solver  Pro  (Frontline  Systems,  Inc.)  is  an  add-in  to  the  Microsoft  Excel 
spreadsheet  software  that  allows  users  to  set  ranges  and  distributions  on  inputs  of 
Excel-based  models.  The  product  operates  similarly  to  Oracle’s  Crystal  Ball  and 
GoldSim  Technology  Group’s  GoldSim  Pro  add-ins.  After  users  establish  the  in¬ 
put  distributions  and  the  number  of  trials,  histograms  and  simulation  statistics  are 
visible  for  review  within  Microsoft  Excel. 

Robust  Portfolio  Management  (RPM)  Decisions  (SAL,  Aalto  University)  is  a 

web-based  Java  tool  for  analyzing  multicriteria  portfolio  problems.  Users  provide 
tabular  data  and  enter  the  alternatives,  attributes,  value  measures,  and  weights.  The 
tool  provides  multiple  built-in  algorithms  and  visualizations,  and  supports  interac¬ 
tive  decision  support.  The  University  provides  limited  help  and  limited  documenta¬ 
tion  on  their  web  pages.  Their  web  pages  are  confusing,  have  poor  documentation, 
and  it  is  unclear  how  it  fits  into  a  larger  suite  of  capabilities.  RPM  is  a  part  of  the 
Decisionarium  site  for  interactive  multicriteria  decision  support  tools. 

Salford  Predictive  Modeler  (Salford  Systems)  is  a  software  suite  that  contains  the 
CART,  MARS,  TreeNet,  and  RandomEorests  tools.  The  suite  essentially  performs 
the  combined  functions  of  the  individual  tools  that  comprise  it,  including  data  min¬ 
ing  and  regression,  decision  trees,  and  model  construction.  Salford  Systems  offers 
online  and  in  person  training  and  support. 

Sensit  (TreePlan  Software)  is  a  sensitivity  analysis  add-in  for  the  Microsoft  Excel 
spreadsheet  software.  Sensit  performs  sensitivity  analysis  on  existing  Excel-based 
models.  TreePlan  provides  answers  to  frequently  asked  questions  for  their  software. 

ServiceModel  (ProModel  Corporation)  is  a  planning  and  throughput  analysis  tool 
for  the  service  industry.  The  tool  can  analyze,  visualize,  and  optimize  the  layout  of 
a  service  facility  based  on  performance  metrics  specified  by  the  user.  Users  can 
also  pose  what-if  questions  about  the  layout  and  its  constraints,  and  the  tool  will 
provide  options  for  improving  the  performance  metrics.  The  tool  has  import/export 
capabilities  with  the  Microsoft  Excel  spreadsheet  software.  ProModel  Corporation 
provides  online  and  classroom  based  training  programs  and  provides  online  support. 

SIMULIA  (Dassault  Systemes)  is  a  commercial  software  brand  of  simulation  tools 
from  Dassault  Systemes  that  includes  Abaqus  PEA,  fe-safe,  Isight,  Simulation  Eife- 
cycle  Management,  Tosca,  and  two  CATIA  environment  packages.  The  CATIA 
packages,  named  SIMUEIA  V5  and  SIMUEIA  V6,  provide  integrated  analysis  ca- 
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pabilities  that  allow  for  direct  execution  from  within  CATIA.  The  SIMULIA  family 
of  software  provides  a  complete  design,  analysis,  optimization,  visualization,  and 
automation  environment.  Dassault  Systemes  offers  training  courses,  documenta¬ 
tion,  certification  courses,  and  online  and  phone  support. 

Smart-Swaps  (SAL,  Aalto  University)  is  a  method  and  tool  to  reduce  the  number 
of  criteria  and  alternatives  for  evaluation.  The  method  is  based  on  the  book  by 
Hammond  et  al.^  The  University  provides  limited  help  and  limited  documentation 
on  their  web  pages.  Their  web  pages  are  confusing,  have  poor  documentation,  and 
it  is  unclear  how  it  fits  into  a  larger  suite  of  capabilities.  Smart-Swaps  is  a  part  of 
the  Decisionarium  site  for  interactive  multicriteria  decision  support  tools. 

STATISTICA  (StatSoft)  is  a  commercial  comprehensive  statistics  and  analytics 
software  suite  developed  by  StatSoft.  It  operates  on  the  Microsoft  Windows  oper¬ 
ating  system  and  is  available  in  web-based  and  desktop  form.  StatSoft  is  a  direct 
competitor  to  the  SAS’s  Analytics  as  it  provides  similar  capabilities  in  data  mining, 
predictive  modeling,  statistical  analysis,  and  data  visualization.  STATISTICA  can 
be  extended  through  VBA  scripting  and  integration  with  programs  coded  in  R  lan¬ 
guage.  StatSoft  offers  online  support  and  training  along  with  free  to  access  videos 
and  tutorials. 

System  Capabilities  Analytic  Process  (SCAP)  (ARL  Survivability/Lethality  Analysis 
Directorate  [SLAB])  is  a  method  that  maps  the  relationship  between  components 
of  a  system  and  the  system’s  capabilities,  creating  what  is  called  the  functional 
skeleton  (FS).  The  FS  will  depict  what  system  capabilities  are  functional  and  which 
system  components  are  critical  to  the  system’s  primary  goals.  This  allows  for  recog¬ 
nition  of  loss  of  mission  tasks  or  system  capabilities  when  components  of  a  system 
fail. 

Systems  Platform  for  Integrated  Design  in  Realtime  (SPIDR)  (University  of 
Southern  California)  is  a  tool/method  developed  by  the  Institute  for  Systems  Inte¬ 
gration  at  the  University  of  Southern  California.  SPIDR  is  programmed  in  a  variant 
of  C.  SPIDR  is  an  artificial  intelligence  based  search  and  optimization  engine,  also 
called  a  "constraint-based  design  synthesis  engine".  SPIDR  can  automatically  se¬ 
lect  from  a  set  of  existing  components,  each  with  an  array  of  attributes,  to  build  the 
most  preferred  system  for  meeting  decision  maker  preferences  for  values  of  mission 

^Hammond  JS,  Keeney  RL,  Raiffa  H.  Smart  choices:  a  practical  guide  to  making  better  decisions. 
Boston  (MA):  Harvard  Business  Review  Press;  1998. 
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metrics.  SPIDR  can  generate  multiple  designs,  which  are  then  modeled,  analyzed, 
and  simulated  to  document  performance  capabilities.  There  is  an  optimization  algo¬ 
rithm,  however  details  of  this  algorithm  are  unknown.  The  user  builds  XML-based 
hierarchies  within  SPIDR  that  are  called  upon  during  optimization.  Limited  details 
are  available  on  how  optimization  is  accomplished,  and  what  the  output  data  looks 
like  (or  is  explored). 

Systems  Analysis  and  Experimentation  (SAE)  (MITRE)  is  a  framework  for  exe¬ 
cuting  simulation,  optimization,  analysis,  and  visualization  commensurate  with  the 
time  frame  and  confidence  level  requested  by  a  sponsor.  Within  this  framework, 
there  are  several  visualization  tools  of  interest,  including  SpiderView  and  Chart- 
Builder,  which  enable  continuous  and  immersive  data  analysis  and  visualization. 
The  entire  systems  engineering  framework  has  been  under  development  for  some 
time  by  MITRE  and  portions  have  been  applied  to  real-world  scenarios.  The  types 
of  tools  within  the  SAE  set  include  visualization,  decision  analysis,  and  M&S,  so  it 
is  difficult  to  categorize  this  toolset  within  a  single  tradespace  exploration  function. 

Tableau  (Tableau  Software)  is  a  data  visualization  tool,  primarily  used  by  the  busi¬ 
ness  intelligence  community.  With  user-provided  data.  Tableau  can  generate  inno¬ 
vative,  visually  striking  plots.  Recently  Tableau  has  added  the  ability  to  interface 
with  R,  such  that  users  can  perform  all  of  their  statistical  data  analysis  with  R  and 
then  open  that  data  in  Tableau  for  visualization.  Tableau  Software  offers  training, 
white  papers,  blogs,  and  online  tutorials. 

Technology  Identification,  Evaluation,  and  Selection  (TIES)  (Georgia  Institute 
of  Technology)  is  a  method  for  assessing  trade-off  impact  of  technologies  with¬ 
out  time-consuming  mathematical  formulations.  Technically  feasible  solutions  are 
identified  with  accuracy  and  speed  to  reduce  design  cycle  time  and  costs  through 
the  use  of  DOE  and  probabilistic  methods  such  as  RSEs  and  Monte  Carlo  simula¬ 
tion.  In  this  sequential  step  method  the  design  space  is  defined,  systems  are  modeled 
and  simulated,  and  solutions  are  analyzed  and  their  sensitivities  to  technologies  are 
assessed.  Through  the  use  of  IMP,  several  data  visualizations  are  used  to  aid  the 
decision  maker  in  technology  selection. 
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Tradespace  Analysis  for  Capabilities,  Effectiveness,  and  Resources  (TRACER) 
(Logistics  Management  Institute)  is  a  process  for  linking  design  parameters  with 
effectiveness  metrics  and  cost.  In  this  way,  a  user  can  change  inputs  to  determine  the 
cost  and  performance,  or  set  the  cost  and  performance  to  determine  the  design  pa¬ 
rameter  settings  to  achieve  them.  TRACER  attempts  to  optimize  within  and  across 
decision-making  levels,  from  system  design  and  optimization  (What  is  the  best  de¬ 
sign?),  through  concept  analysis  and  selection  (Which  solution  approach  is  best 
for  a  mission?),  and  to  mission,  strategy,  and  force  assessment  (What  is  the  best 
portfolio?).  TRACER  depends  on  cost  and  effectiveness  models  linked  to  design 
parameters,  with  weighted  sums  determining  the  preferred  course  of  action.  There 
are  no  publications  or  references  available  since  the  initial  2007  National  Defense 
Industrial  Association  presentation  (http://www.dtic.mil/ndia/2007test/Belcher_Se 
ssionDl.pdf),  and  inquiries  to  the  presenters  have  gone  unanswered. 

TreeNet  (Salford  Systems)  is  a  commercial  data  mining  tool.  Its  strength  is  the 
capability  to  regress  models  by  generating  "trees"  that  represent  changes  in  response 
given  known  impacts  to  the  response  by  changes  in  other  variables.  TreeNet  can 
assess  the  strength  of  classification,  binary,  and  multiclass  problems  using  statistical 
analysis.  Salford  Systems  offers  online  and  in  person  training  and  support. 

TreePlan  (TreePlan  Software)  is  a  decision  tree  add-in  for  the  Microsoft  Excel 
spreadsheet  software.  TreePlan  calculates  values  of  each  decision  tree  branch  us¬ 
ing  existing  Excel-based  models.  TreePlan  provides  answers  to  frequently  asked 
questions  for  their  software. 

TreePlan  ToolKit  (TreePlan  Software)  is  the  bundle  of  all  3  TreePlan  Software 
add-ins  for  the  Microsoft  Excel  spreadsheet  software:  TreePlan,  Sensit,  and  RiskSim. 
TreePlan  provides  answers  to  frequently  asked  questions  for  their  software. 

VisLab  (Massachusetts  Institute  of  Technology  [MIT])  is  an  internally  developed 
research  testbed  for  analysis  techniques  and  visualizations.  VisEab  is  MATEAB- 
based,  possibly  similar  in  nature  to  Georgia  Tech’s  Rave.  The  tool  and  accompany¬ 
ing  process  were  developed  at  MIT. 

Visual  Design  &  Optimization  Control  (VisualDOC)  (Vanderplaats  R&D,  Inc.) 

is  a  general  purpose  multidisciplinary  design,  optimization,  and  process  integration 
software.  Analysis  modules  can  be  added  to  it  (DOE,  ES-DYNA,  finite  element 
analysis  (EEA),  CED).  VisualDOC  can  output  data  in  parallel  plots  and  2-D/3-D 
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scatterplots.  The  layout  of  VisualDOC  is  similar  to  ModelCenter,  where  the  user 
ean  ereate  a  flowehart  of  inputs,  outputs,  and  eomponents,  and  link  eodes  together 
(primarily  for  export  to  Mierosoft  Exeel  spreadsheet  software  and  MATLAB,  but 
also  to  external  tools  using  wrappers).  VisualDOC  allows  for  automation  of  the 
design  and  optimization  proeess.  Built-in  eomponents  inelude  optimization,  DOE, 
response  surfaee  methodology,  and  probabilistie  analysis.  Vanderplaats  offers  sem¬ 
inars,  workshops,  teehnieal  support,  and  training. 

Web-HIPRE  (SAL,  Aalto  University)  is  a  web-based  deeision  support  tool  based 
on  multi- attribute  value  theory  weighting  teehniques  and  analytie  hierarehy  proeess. 
This  is  a  web  version  of  the  HIPRE  3-i-  deeision  analysis  software.  The  University 
provides  limited  help  and  limited  doeumentation  on  their  web  pages.  Their  web 
pages  are  eonfusing,  have  poor  doeumentation,  and  it  is  unelear  how  it  fits  into 
a  larger  suite  of  eapabilities.  Web-HIPRE  is  a  part  of  the  Deeisionarium  site  for 
interaetive  multieriteria  deeision  support  tools. 

Whole  System  Trades  Analysis  Tool  (WSTAT)  (Program  Executive  Office  Ground 
Combat  Systems  (PEO-GCS)  and  Sandia  National  Laboratories)  is  the  Java 
version  of  AAMODAT.  AAMODAT  was  provided  by  PEO-GCS  to  Sandia  Eabs 
to  be  enhaneed  for  inelusion  with  CPAT  as  a  system-level  optimization  tool.  The 
WSTAT  exeeutable  is  government-owned,  but  the  souree  eode  rights  belong  to  San¬ 
dia.  Output  from  WSTAT  is  intended  to  feed  into  CPAT.  In  this  way,  WSTAT  op¬ 
timizes  a  system  aeross  5  dimensions  (unit  eost,  O&S  eost,  performanee,  growth, 
and  risk)  and  CPAT  optimizes  a  portfolio  aeross  performanee,  sehedule,  and  eost. 
WSTAT  is  intended  for  use  at  the  program  manager  level,  whereas  CPAT  is  intended 
for  use  at  the  Program  Exeeutive  Offlee  level. ^ 

WSTAT  is  essentially  identieal  to  AAMODAT — requirements  getting  mapped  to 
the  funetional  objeetives,  the  funetional  objeetives  getting  mapped  to  the  physieal 
arehiteeture,  teehnology  options  affeeting  system  metries  (whieh  traee  baek  to  the 
physieal  arehiteeture),  value  funetions  deseribing  stakeholder  value  over  the  range 
of  possible  metrie  measures,  and  priority  weightings  deseribing  the  importanee  of 
the  metrie  to  the  stakeholder.  The  outputs  ean  be  visualized  in  seatterplot,  spider 
ehart,  tornado  plot,  and  histogram  formats.  In  this  way,  depending  upon  the  pri¬ 
ority  weighting  of  the  system  funetions,  the  appropriate  subsystems  are  eombined 

^Edwards  S.  Capability  portfolio  analysis  tool  (CPAT)  &  whole  system  trade  analysis  (WSTA). 
Paper  presented  at:  Army  Systems  Engineering  Eorum  (ASEE).  2013  Eeb  27;  Arlington,  VA. 
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to  provide  the  highest  value  to  the  stakeholder.  There  are  a  few  drawbacks  to  the 
WSTAT  tool,  in  that  multiobjective  optimization,  multiattribute  utility,  uncertainty, 
and  sensitivities  are  not  addressed. 
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Appendix  B.  TSE  Tool  Keep/Reject  Rationale 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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TSE  Tool 

Developer 

Keep/ 

Reject 

Now/ 

Follow- 

On 

Rationale  for  Keep  /  Reject 

(BRISK 

Palisade 

Corporation 

Reject 

Neither 

Included  in  DecisionTools  Suite  6.0. 

lOOOMinds 

lOOOMinds  Ltd 

Reject 

Neither 

Web  site  is  no  longer  available.  No  other  detailed 
information  available. 

AAMODAT 

RDECOM- 

ARDEC 

Keep 

Now 

Development  continues  within  DoD  for  TSE.  Capability  has 
been  spun  off  into  WSTAT,  which  is  now  packaged  with 

CPAT. 

AgenaRisk 

Agena  Limited 

Keep 

Follow- 

on 

Performs  risk  assessment,  probabilistics,  sensitivity  analysis, 
and  visualization  to  solve  complex  risk  problems.  Further 
investigation  needed  for  follow-on  assessment  to  determine 
depth  of  applicability  to  TSE. 

Analytica 

Lumina 

Decision 

Systems 

Keep 

Follow- 

on 

Similar  in  practice  to  Excel,  but  establishes  systems  of 
equations  in  a  visual  (flowchart)  style.  Capable  of  uncertainty 
assessment  and  probabilistic  analysis.  Insufficient  knowledge 
of  the  tool  for  consideration  in  current  effort. 

Analytics 

SAS 

Keep 

Follow- 

on 

Suite  of  integrated  statistical  analysis  tools.  This  is  the  entire 
SAS  tool  portfolio.  Needs  more  time  for  further  assessment. 

ASEC 

RDECOM- 

TARDEC 

Keep 

Now 

Continued  development  within  DoD  for  TSE.  Web-based, 
government-developed  framework.  Decision-centered 
framework  for  assessing  risk,  requirements,  and  capabilities 
from  a  system  functional  perspective. 

ATSV 

Penn  State  ARL 

Keep 

Now 

Continued  development  within  DoD  for  TSE.  Multi¬ 
dimensional  visualization  tool. 

Berkeley 

Madonna 

University  of 
California  at 
Berkeley 

Reject 

Neither 

A  differential  equation  solver.  Provides  a  specific,  limited 
application. 

CART 

Salford  Systems 

Reject 

Neither 

Classification  and  regression  decision  tree  tool;  a  part  of  the 
Salford  Predictive  Modeler  suite. 

ClearPoint 

Strategy 

Ascendant 

Strategy 

Management 

Group 

Reject 

Neither 

A  strategy  management  tool.  Maps  strategic  objectives  and 
measures  to  corporate  activities.  Not  applicable  to  TSE. 

Comparion 

Suite 

Expert  Choice, 
Inc. 

Reject 

Neither 

Limited  information  from  developer.  Strictly  performs 
portfolio  and  organizational  decision  analysis.  Not  applicable 
to  TSE. 

CPAT 

PEO-GCS 

Keep 

Now 

Continued  development  within  DoD  for  TSE.  Portfolio 
analysis  tool. 

CyDesign 

Studio 

CyDesign  Labs 

Keep 

Follow- 

on 

Similar  in  layout  to  FACT.  A  model-based  systems 
engineering  tool  that  looks  at  complex  systems  holistically. 
Insufficient  knowledge  of  the  tool  for  consideration  in 
current  effort. 

D3 

D3JS.org 

Reject 

Neither 

Specific  to  visualization.  Relies  on  external  data  tables. 
Enhances  TSE  tools,  but  is  not  itself  a  TSE  tool. 

DAKOTA 

Sandia  National 

Labs 

Keep 

Follow- 

on 

Object-oriented  framework  for  design  optimization, 
parameter  estimation,  uncertainty  quantification,  and 
sensitivity  analysis.  Need  to  assess  in  depth  in  a  follow-on 
effort. 

DEA  SolverPro 

SAITECH,  inc. 

Reject 

Neither 

Data  envelopment  analysis;  Limited  documentation  from 
website.  Not  applicable  to  TSE. 

DecidelT 

Preference 

Keep 

Follow- 

on 

Decision  analysis  tool  that  can  handle  imprecision  and 
uncertainty.  Insufficient  knowledge  of  the  tool  for 
consideration  in  current  effort. 

Decision 

Explorer 

Banxia 

Software  Ltd 

Reject 

Neither 

Helps  capture  thoughts  and  ideas  in  detail  (like  sticky  notes). 
Limited  applicability  to  TSE. 
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TSE  Tool 

Developer 

Keep/ 

Reject 

Now/ 

Follow- 

On 

Rationale  for  Keep  /  Reject 

DecisionTools 

Suite  6.0 

Palisade 

Corporation 

Keep 

Follow- 

on 

Excel  add-on  that  conducts  risk  analysis  under  uncertainty, 
statistical  analysis,  and  data  visualization.  Further 
investigation  needed  for  follow-on. 

D-Sight 

Desktop 

D-Sight 

Keep 

Follow- 

on 

Provides  a  framework  for  multi-stakeholder,  multi-criteria 
decision  analysis.  Follows  a  structured  decision  analysis 
process.  Has  data  visualization  capabilities,  but  looks  to  be 
mainly  focused  on  capturing  value  and  developing  utility 
functions.  Insufficient  knowledge  of  the  tool  for 
consideration  in  current  effort. 

Enterprise 

Portfolio 

Simulator 

ProModel 

Corporation 

Reject 

Neither 

High-level  strategic  planning,  project  and  portfolio 
management  tool.  A  collaborative  and  web-based  version  of 
Portfolio  Simulator. 

EquityS 

Catalyze  Ltd 

Reject 

Neither 

Similar  to  the  HiviewB  product,  but  with  specific  application 
to  portfolio  management  decisions. 

Eureqa 

Desktop 

Nutonian,  Inc. 

Reject 

Neither 

Capability  resides  in  larger  decision  analysis  or  numerical 
analysis  tools  already  under  consideration.  However,  the  tool 
is  free  to  download  and  use. 

Excel 

Microsoft 

Keep 

Now 

Most  common  visualization  and  analysis  tool.  Ubiquitous. 

ExtendSim 

Suite 

Imagine  That, 
Inc. 

Keep 

Follow- 

on 

Model-based  simulation  tool  for  continuous  and  discrete 
processes.  Visualization  and  statistical  analysis  features  are 
included.  Insufficient  knowledge  of  the  tool  for  consideration 
in  current  effort. 

FACT 

GTRI 

Keep 

Now 

Continued  development  within  DoD  for  TSE. 

ForeTell 

DecisionPath, 

Inc. 

Reject 

Neither 

Project  and  portfolio  management  tool  that  has  some 
analytic  and  data  analysis  capability.  Not  applicable  to  TSE 
for  engineered  systems. 

GoldSim  Pro 

GoldSim 

Technology 

Group 

Reject 

Neither 

Although  apparently  more  capable  at  modeling  discrete  and 
continuous  processes  than  competing  tools  (@RISK,  Crystal 

Ball,  Risk  Solver)  GoldSim  is  essentially  a  discrete  event 
simulator  and  probabilistic  simulation  add-in  for  Microsoft 
Excel.  Limited  application  as  a  probabilistic  simulation  add-in. 

gPROMS 

Process 

Systems 

Enterprise 

Limited 

Keep 

Follow- 

on 

High-fidelity  predictive  model-based  engineering  and 
optimization  software  for  process  industries.  Appears  able  to 
create  custom  physics-based  models  with  ability  to  simulate. 
Insufficient  knowledge  of  the  tool  for  consideration  in 
current  effort. 

GRIPS 

The  Aerospace 
Corp 

Keep 

Follow- 

on 

Decision-support  process  that  uses  high-dimensional 
visualization  to  solve  complex  problems.  Limited 
documentation  available.  Insufficient  knowledge  of  the  tool 
for  consideration  in  current  effort. 

FliviewS 

Catalyze  Ltd 

Reject 

Neither 

Limited  to  decision  analysis  on  multi-stakeholder  input. 
Traditional  decision  analysis  where  alternatives  are  scored 
against  criteria  and  are  ranked  based  on  user  values.  Appears 
similar  to  the  web-based  Decision  Lens  collaborative  tool. 

Iris 

Ayasdi 

Reject 

Neither 

Looks  at  relationships  amongst  variables  in  data.  The  visual 
aspect  is  what  makes  this  an  enticing  TSE  tool,  although  may 
not  be  applicable  to  the  entire  TSE  process. 

Isight 

Dassault 

Systems 

Keep 

Follow- 

on 

Similar  to  ModelCenter.  Insufficient  knowledge  of  the  tool 
for  consideration  in  current  effort. 

JIAT 

ODASA-CE 

Reject 

Neither 

Tool  is  specifically  used  to  build  up  cost  estimates  and  is  not 
applicable  to  TSE. 

JMP 

SAS 

Keep 

Now 

Statistics  with  interactive  graphics;  excels  at  visualization, 
surrogate  modeling,  and  statistical  analysis. 
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TSE  Tool 

Developer 

Keep/ 

Reject 

Now/ 

Follow- 

On 

Rationale  for  Keep  /  Reject 

KNITRO 

optimization 

solver 

Ziena 

Optimization 

LLC 

Reject 

Neither 

Limited  application  as  a  non-linear  optimization  solver. 

Logical 
Decisions  for 

Windows 

Logical 

Decisions 

Reject 

Neither 

Similar  logical  analysis  process  as  AAMODAT,  but  does  not 
rank  or  optimize.  Part  of  the  Logical  Decisions  Portfolio 
product.  A  decision  analysis  product  but  marketed  as  a 
portfolio  management  product. 

Logical 

Decisions 

Portfolio 

Logical 

Decisions 

Keep 

Follow- 

on 

Based  on  the  Logical  Decisions  for  Windows  product. 
Implements  optimization  and  selection  of  multiple 
alternatives  based  on  constraints.  A  decision  analysis  product 
but  marketed  as  a  portfolio  management  product. 

Insufficient  knowledge  of  the  tool  for  consideration  in 
current  effort. 

MapleSim 

Maplesoft 

Keep 

Follow- 

on 

Similar  to  Simulink.  Built  from  the  open  source  Modelica 
library.  Insufficient  knowledge  of  the  tool  for  consideration 
in  current  effort. 

MARS 

Salford  Systems 

Reject 

Neither 

Limited  application  as  a  regression  tool.  Part  of  the  Salford 
Predictive  Modeler  suite. 

Mathematica 

Wolfram 

Keep 

Follow- 

on 

Widely  used  computational  environment  used  for  modeling, 
analyses,  programming,  and  visualization.  Further 
investigation  needed  for  follow-on. 

Matlab 

Mathworks 

Keep 

Now 

Common  numerical  analysis  tool,  large  user  community, 
currently  in  use  by  ERS  Demo  team;  ARL  familiarity 

MedModel 

Optimization 

Suite 

ProModel 

Corporation 

Reject 

Neither 

Limited  application  to  medical  facility  modeling,  planning, 
and  evaluation. 

modeFRONTIER 

Esteco 

Keep 

Follow- 

on 

Multidisciplinary  analysis  and  optimization  tool  used  to  link 
outside  analysis  tools  and  provide  some  statistical  analysis 
and  post  processing.  Similar  to  ModelCenter.  Insufficient 
knowledge  of  the  tool  for  consideration  in  current  effort. 

ModelCenter 

Phoenix 

Integration 

Keep 

Now 

Common  MDAO  tool,  moderately  sized  user  community, 
currently  in  use  by  ERS  Demo  team. 

Nexus 

iChrome  Ltd. 

Keep 

Follow- 

on 

Similar  functionality  and  scope  as  ModelCenter  and 
modeFRONTIER.  Insufficient  knowledge  of  the  tool  for 
consideration  in  current  effort. 

OpenMDAO 

NASA 

Keep 

Now 

Free,  open  source  MDAO  tool  developed  by  NASA  Glenn. 
Gaining  popularity  in  government  research  community. 

OptiY 

OptiY  GmbH 

Keep 

Follow- 

on 

Similar  to  ModelCenter  but  may  have  more  statistical 
functionality.  Insufficient  knowledge  of  the  tool  for 
consideration  in  current  effort. 

Portfolio 

Simulator 

ProModel 

Corporation 

Reject 

Neither 

Not  a  TSE  tool,  strictly  looking  at  portfolio  optimization 

Process 

Simulator 

ProModel 

Corporation 

Reject 

Neither 

MS  Visio  plug-in  for  process  analysis  and  optimization;  not 
standalone  tool,  other  software  required  for  use. 

Project 

Simulator 

ProModel 

Corporation 

Reject 

Neither 

MS  Project  plug-in  for  schedule  analysis  and  optimization; 
not  standalone  tool,  other  software  required  for  use. 

Promax 

Professional 

Cogentus 

Consulting 

Limited 

Keep 

Follow- 

on 

Aids  in  conducting  a  detailed  decision  analysis  process. 
Insufficient  knowledge  of  the  tool  for  consideration  in 
current  effort. 

ProModel 

Optimization 

Suite 

ProModel 

Corporation 

Reject 

Neither 

Modeling  of  continuous  processes  for  optimization  of 
planning,  warehousing,  and  logistics. 

R 

R  Foundation 

Keep 

Now 

Widely  used  statistical  analysis  focused  programming 
language.  Top  in  Rexer  surveys. 
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TSE  Tool 

Developer 

Keep/ 

Reject 

Now/ 

Follow- 

On 

Rationale  for  Keep  /  Reject 

RandomForests 

Salford  Systems 

Reject 

Neither 

Decision  tree  tool;  a  part  of  the  Salford  Predictive  Modeler 
suite. 

Rave 

Georgia 
Institute  of 
Technology 

Keep 

Follow- 

on 

Freeware  TSE  tool  developed  in  Matlab  (a  commonly  used 
tool).  Similar  functionality  to  JMP.  Further  investigation 
needed  for  follow-on. 

RICH  Decisions 

Systems 
Analysis 
Laboratory, 
School  of 
Science,  Aalto 
University 

Reject 

Neither 

An  alternative  ranking  and  preference  modeling  tool  that  is 
included  in  the  web-based  RPM-Decisions  tool.  Confusing 
web  page,  poor  documentation,  and  unclear  how  it  fits  into  a 
larger  suite  of  capabilities. 

Risk  Solver  Pro 

Frontline 
Systems,  Inc. 

Reject 

Neither 

Excel  add-in  for  performing  probabilistic  Monte  Carlo 
simulation  on  Excel-based  models.  Similar  in  functionality  to 
GoldSim.  Limited  application  as  a  probabilistic  simulation 
add-in. 

RiskSim 

TreePlan 

Software 

Reject 

Neither 

Included  in  TreePlan  Toolkit.  An  MS  Excel  add-in  for  Monte 
Carlo  simulations  for  Excel  based  models,  random  number 
generator,  and  plotter. 

ROSETTA 

Georgia 
Institute  of 
Technology 

Reject 

Neither 

Limited  application  to  capturing  stakeholder  value.  Although 
the  tool  has  a  novel  approach  to  mapping  stakeholder  needs 
to  value,  it  is  not  in  itself  a  TSE  tool.  Could  be  useful  as  part 
of  a  piecemeal  TSE  environment. 

RPM-Decisions 

Systems 
Analysis 
Laboratory, 
School  of 
Science,  Aalto 
University 

Keep 

Follow- 

on 

A  Java  software  tool  that  performs  portfolio  modeling  and 
optimization  using  MCDA.  However,  could  be  implemented 
to  perform  analysis  of  alternative  designs  within  a 
tradespace.  Confusing  web  page,  poor  documentation,  and 
unclear  how  it  fits  into  a  larger  suite  of  capabilities. 

Insufficient  knowledge  of  the  tool  for  consideration  in 
current  effort. 

SAE 

MITRE 

Keep 

Follow- 

on 

A  wide  ranging  set  of  systems  engineering  tools,  including 
visualization,  M&S,  and  decision  analysis.  Insufficient 
knowledge  of  the  tool  for  consideration  in  current  effort. 

Salford 

Predictive 

Modeler 

Salford  Systems 

Keep 

Follow- 

on 

This  is  the  suite  which  contains  the  other  four  Salford 
decision  tree  and  regression  products:  CART,  TreeNet,  MARS, 
and  RandomForests.  Further  investigation  needed  for  follow- 

on. 

SCAP 

RDECOM-ARL- 

SLAD 

Keep 

Follow- 

on 

A  government-developed  and  owned  tool.  Can  be  used  to 
assess  interaction  between  subsystems  and  components  and 
the  whole  system's  cost  or  performance.  Limited 
documentation  and  use  cases,  but  worthy  of  a  follow-on 
analysis  given  the  model-based  structure  and  government 
ownership. 

Sensit 

TreePlan 

Software 

Reject 

Neither 

Included  in  TreePlan  Toolkit.  An  MS  Excel  add-in  for 
performing  a  sensitivity  analysis. 

ServiceModel 

Optimization 

Suite 

ProModel 

Corporation 

Reject 

Neither 

Specific  to  the  service  industry,  specifically  looking  at 
throughput  analysis. 

SIMULIA 

Dassault 

Systemes 

Reject 

Neither 

Specific  to  CAD  models  and  FEA.  Not  applicable  to  TSE. 

Smart-Swaps 

Systems 
Analysis 
Laboratory, 
School  of 
Science,  Aalto 
University 

Reject 

Neither 

A  method  and  tool  to  reduce  the  number  of  criteria  and 
alternatives  for  evaluation.  Confusing  web  page,  poor 
documentation,  and  unclear  how  it  fits  into  a  larger  suite  of 
capabilities. 
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TSE  Tool 

Developer 

Keep/ 

Reject 

Now/ 

Follow- 

On 

Rationale  for  Keep  /  Reject 

SPIDR 

University  of 
Southern 

California 

Keep 

Follow- 

on 

A  tool/method  that  uses  Al-driven  search  and  optimization. 
Uses  XML-based  system  hierarchies  to  allocate  components 
in  order  to  meet  user  objectives.  Limited  documentation  and 
use  cases,  but  worthy  of  a  follow-on  analysis  given  that  it  is 
marketed  as  a  TSE  tool. 

STATISTICA 

StatSoft 

Keep 

Follow- 

on 

High  percentage  of  users  from  Rexer  survey  and  high  user 
satisfaction.  Similar  in  role  to  SAS.  Further  Investigation 
needed  for  follow-on. 

Tableau 

Tableau 

Software 

Keep 

Follow- 

on 

Data  mining  and  visualization  tool.  Ease  of  use,  customizable 
dashboards.  Insufficient  knowledge  of  the  tool  for 
consideration  in  current  effort. 

TIES 

Georgia 
Institute  of 
Technology 

Keep 

Now 

The  techniques  used  within  the  method,  and  the  method 
overall,  is  directly  applicable  to  TSE.  A  structured  process  for 
assessing  the  impact  of  adding  technologies  to  systems, 
which  can  be  compared  with  FACT. 

TRACER 

LMI  Consulting 

Reject 

Neither 

Limited  publications;  no  apparent  usage  since  initial  NDIA 

2007  presentation. 

TreeNet 

Salford  Systems 

Reject 

Neither 

Decision  tree  tool;  a  part  of  the  Salford  Predictive  Modeler 
suite. 

TreePlan 

TreePlan 

Software 

Reject 

Neither 

Included  in  TreePlan  ToolKit;  MS  Excel  add-in  for  creating 
decision  trees. 

TreePlan 

ToolKit 

TreePlan 

Software 

Keep 

Follow- 

on 

Toolkit  which  includes  a  decision  tree  designer,  sensitivity 
assessment  tool,  and  Monte  Carlo  simulation  tool.  For  use 
with  Excel  based  models.  Further  investigation  needed  for 
follow-on;  website  is  relatively  non-descriptive,  and  it 
appears  that  these  tools  have  focused  and  limited 
functionality. 

VisLab 

MIT 

Keep 

Follow- 

on 

Use  currently  limited  to  internal  MIT.  Enables  preference- 
driven  TSE.  Limited  documentation  and  use  cases,  but 
worthy  of  a  follow-on  analysis. 

VisualDOC 

Vanderplaats 
R&D,  Inc. 

Keep 

Follow- 

on 

Similar  to  ModelCenter.  Performs  DOE,  RSM,  and 
probabilistics.  Covers  several  aspects  of  TSE  processes. 
Insufficient  knowledge  of  the  tool  for  consideration  in 
current  effort. 

Web-HIPRE 

Systems 
Analysis 
Laboratory, 
School  of 
Science,  Aalto 
University 

Reject 

Neither 

A  web-based  value  tree  and  AHP  decision  support  tool. 
Confusing  web  page,  poor  documentation,  and  unclear  how 
it  fits  into  a  larger  suite  of  capabilities. 

WSTAT 

PEO-GCS 

Keep 

Now 

Continued  development  within  DoD  for  TSE. 
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Appendix  C  TSE  Tool  Attribute  Definition  and  Rationale 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Pricing  Information  Operating  Systems  Usage  Class 


Attribute 

Definition 

Rationale  for  Inclusion  Primary  Origin 

Tool 

A  piece  of  software,  proprietary  or 
commercial,  to  conduct  tradespace 
exploration,  analysis,  and/or  data 
visualization. 

Users  need  to  differentiate 
between  the  application  of 
a  tool  and  a  process.  There 
will  be  cases  where  the  tool 

is  embedded  within  a 

process. 

ARL 

Process 

An  analysis  method  or  workflow  for 
conducting  tradespace  exploration, 
analysis,  and/or  data  visualization. 

Users  need  to  differentiate 
between  the  application  of 
a  tool  and  a  process.  There 
will  be  cases  where  the  tool 

is  embedded  within  a 

process. 

ARL 

Specific  industries  or 
market  segments  in 
which  the  software 
tool  is  more  popular 

Are  there  sectors  of  industry  that 
routinely  use  the  tool  most  often? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Specific  applications 
in  which  the 

software  tool  is  more 
widely  used  for 

What  is  the  specific  task  or  application 
that  the  tool  is  used  for  most  often? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

User  or  developer 
community  size 

How  large  and  mature  is  the  community 
who  develops,  uses,  supports,  and 
advances  this  tool? 

It  is  important  to  know  if 
the  developer  community  is 
small  or  large,  or  if  it  is 
growing  or  shrinking.  Want 
to  avoid  investing  in  the 
current  "best  thing"  and 
instead  place  investments 
on  tools  that  are  on  the 
upswing. 

2011  Rexer  Data 
Miner  Survey 

Windows 

Does  the  tool  operate  in  Windows 
operating  system? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Mac  OS 

Does  the  tool  operate  in  Macintosh 
operating  system? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Unix/Linux 

Does  the  tool  operate  in  Unix  or  Linux 
operating  systems? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Flavors  of  Unix/Linux 

Which  versions  of  Unix  or  Linux 
operating  systems  can  the  tool  operate 
in? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Web  implementation 

Can  the  tool  be  implemented  or  used 
from  the  open  web  (e.g.,  http  or  https)? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Other  Operating 
Systems 

Can  the  tool  operate  in  operating 
systems  not  listed  above? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Commercial 

What  is  the  purchase  price  of  the 
standard  commercial  version  of  the 
tool/process? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Education 

What  is  the  purchase  price  of  the 
education  version  of  the  tool/process? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Enhanced/High 

Performance 

What  is  the  purchase  price  of  the 
enhanced  or  higher  performance  (i.e., 
'deluxe')  version  of  the  tool/process? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Attribute 

Definition 

Rationale  for  Inclusion 

Primary  Origin 

1 

Open/Free 

Is  the  tool/process  considered  Open 

Source  or  Freeware  software;  available 
without  cost  to  anyone,  public  or 
government? 

There  is  a  growing  interest 
in  tools  that  are  open 
source,  as  well  as  tools  that 
are  free/limited  in 
capability. 

ARL 

Vendor 

Does  the  vendor  offer  training  classes  for 

Legacy  OR/MS  Today  DAS 

2012  OR/MS 

■o 

the  tool/process? 

Survey  Entry 

Today  DAS  Survey 

OJ 

3rd  Party 

Does  a  third  party  organization  offer 

Legacy  OR/MS  Today  DAS 

2012  OR/MS 

O 

(/) 

training  classes  for  the  tool/process? 

Survey  Entry 

Today  DAS  Survey 

OJ 

(/) 

(/) 

ra 

Classroom 

Are  in-classroom  training  sessions 

Legacy  OR/MS  Today  DAS 

2012  OR/MS 

bA 

_c 

offered  by  the  vendor  or  a  third  party? 

Survey  Entry 

Today  DAS  Survey 

*?5 

1- 

Online 

Are  online  training  sessions  offered  by 

Legacy  OR/MS  Today  DAS 

2012  OR/MS 

the  vendor  or  a  third  party? 

Survey  Entry 

Today  DAS  Survey 

A  cursory  level  of 
competency  proficiency 
needs  to  be  assigned.  Users 
need  to  have  an 
understanding  of  the 

Requires  little  to  no  prior  knowledge  of 

expected  relative  level  of 

the  software  interface  or  underlying 

capability  required  to 

Basic 

math  to  effectively  use  and  generate 

effectively  use,  and 

ARL 

results.  Operation  is  intuitive  and  the 

potentially  improve,  the 

user  can  learn  how  to  use  it  with  relative 

tool/process.  The  basic. 

ease. 

intermediate,  and  advanced 

■o 

OJ 

levels  should  be  enough  to 

■o 

OJ 

inform  the  user  of  the 

z 

amount  of  training  and 

OJ 

u 

experience  needed  to 

.2 

conduct  TSE  using  the  tool. 

(U 

a 

A  cursory  level  of 

LU 

competency  proficiency 

bA 

c 

needs  to  be  assigned.  Users 

need  to  have  an 

1- 

Intermediate 

Requires  prior  intermediate  level 
knowledge  or  of  the  subject  matter  to 
effectively  use  the  tool,  generate  and 
interpret  results.  Minimal  training  may 
be  needed. 

understanding  of  the 
expected  relative  level  of 
capability  required  to 
effectively  use,  and 
potentially  improve,  the 
tool/process.  The  basic, 
intermediate,  and  advanced 
levels  should  be  enough  to 
inform  the  user  of  the 
amount  of  training  and 
experience  needed  to 
conduct  TSE  using  the  tool. 

ARL 
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Attribute  | 

1  Definition  | 

1  Rationale  for  Inclusion 

Primary  Origin 

1 

A  cursory  level  of 
competency  proficiency 
needs  to  be  assigned.  Users 
need  to  have  an 

Advanced 

Requires  prior  advanced  level  knowledge 
of  the  subject  matter  to  effectively  use 
the  tool,  generate  and  interpret  results. 
Extensive  training  (possibly  in  the  form 
of  developer  taught  classes)  may  be 
required  to  take  full  advantage  of  all 
capabilities. 

understanding  of  the 
expected  relative  level  of 
capability  required  to 
effectively  use,  and 
potentially  improve,  the 
tool/process.  The  basic, 
intermediate,  and  advanced 
levels  should  be  enough  to 
inform  the  user  of  the 
amount  of  training  and 
experience  needed  to 
conduct  TSE  using  the  tool. 

ARL 

Is  live  or  delayed 
response  help 
available  from  the 
developer  or  broader 
user  community 

Can  the  user  reach  out  to  a  live  person 
for  advice  or  help,  or  is  there  a  user 
forum  that  offers  delayed  responses  to 
questions? 

Beyond  help  menus,  demos, 
or  tutorials,  there  is  a  need 
to  obtain  live  or  delayed 
responses  to  questions, 
inquiries,  issues,  etc.,  as 
well  as  learn  from  others. 

2011  Rexer  Data 
Miner  Survey 

Are  help  menu, 
demos,  or  tutorials 
available 

Is  there  help  available  within  the  tool, 
and  are  there  demo  videos  or  tutorials 
that  can  aid  in  learning  how  to  better 
and  more  fully  use  the  tool? 

Users  need  the  ability  to 
find  answers  to  frequently 
asked  questions,  or  receive 
help  in  executing  a  feature 
of  the  tool. 

2011  Rexer  Data 
Miner  Survey 

Although  this  question  was 
dropped  from  the  2010 
OR/MS  Today  DAS  survey. 

Limitations  on  the 

Are  there  size  (i.e.,  memory,  row. 

this  is  a  concern  for  ERS. 

dataset  capable  of 

column,  etc.)  limitations  to  the  tool's 

There  is  a  need  to 

Sponsor 

(/) 

OJ 

4-* 

being  explored 

capability? 

understand  any  limitations 

3 

n 

of  the  tool  in  terms  of  how 

4-* 

much  data  it  can  analyze. 

< 

plot,  store,  etc. 

ra 

§ 

Not  only  need  to  know  if 

o 

the  tools  is  approved  for 

t/) 

Approved  for  use  on 

Is  the  tool  approved  for  use  on  a 

use  on  a  classified  network, 
but  there  is  also  a  need  to 

Sponsor 

a  classified  network 

classified  network? 

know  what,  if  anything,  is 
preventing  its  use  as  such 
(e.g.,  module  or  algorithm 
written  by  non-US). 

Currently  in  use  by 
any  of  the  ERS 

Is  the  tool  currently  in  use  by  any  of  the 

Desire  to  know  if  the  tool  is 
in  use,  or  planned  for  future 
use,  by  any  of  the  ERS 
Demonstration  Project 

teams. 

Sponsor 

Demonstration 

Project  teams 

ERS  Demonstration  Project  teams? 

Limitations  in  "free" 
software  (e.g., 
toolbox  is  free  but 
host  software  is  not). 

The  tool  may  be  "free"  for 

Are  there  any  limitations  or  caveats  to 
this  tool  being  "free"? 

download  and  use,  but  does 
it  require  a  costly  parent 
tool  (e.g.,  Matlab  required 
to  run  a  specific  tool  box). 

Sponsor 
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Attribute  | 

1  Definition  | 

1  Rationale  for  Inclusion 

Primary  Origin 

Standard  or  open 
data  storage, 
interchange,  and 

reuse 

Does  the  tool  support  standard  file 
formats  for  data  storage,  transfer,  and 
reuse? 

Non-standard  or  proprietary 
data  formats  inhibit  linking 
codes  and  performing 
multi-disciplinary  analysis 
and  optimization. 

ERS  Tradespace 
Workshop 

Primary  Language  to 
Expand  or  Develop 
Functionality 

What  programming  language  allows 
expansion  of  the  tool's  capabilities? 

Include  languages  that  impact  the  native 
tool,  shared  toolboxes,  and  add-ins. 

Knowing  the  programming 
language  will  help  assess 
required  user  proficiency 
level,  compatibility  with 
other  tools,  and  especially 
how  more,  or  custom, 
capabilities  can  be  added. 
Examples  include  Excel, 
Matlab,  and  R. 

ARL 

Applications 

Best  option  using 
multiple  objectives 

Can  the  tool  select  a  best  option  from  a 
list  of  existing  options  using  multiple, 
competing  objectives? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Representation  / 
analysis  of 
uncertainty 

Can  the  tool  represent  uncertainty  or 
perform  analysis  of  the  uncertainty 
within  the  model  results? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Representation  / 
analysis  of 
probabilistic 
dependencies 

Can  the  tool  represent  probabilistic 
dependencies  or  perform  analysis  of  the 
dependencies  between  inputs  and 
outputs? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Risk  Preference 

Can  the  tool  model  risk  preference  of 
individual  stakeholders,  or  of  a  group? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Sequential  decision 
making 

Does  the  tool  conduct  sequential 
decision  making? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Portfolio  decision 
making 

Does  the  tool  conduct  portfolio  decision 
making,  including  project/process 
portfolio  management  and  simulation? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Bayesian  Belief 
Networks 

Can  the  tool  create  and/or  use  Bayesian 
belief  networks? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Multiple 

stakeholders 

collaboration 

Can  the  tool  handle  inputs  from  multiple 
stakeholders,  either  through  combining, 
normalizing,  utility  functions,  or 
preferences? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Software  Features 

Interfaces  to  other 
software: 

Can  the  tool  interface  with  other 
software,  either  manually  or 
automatically,  via  import  or  export  of 
inputs/outputs? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Import  (database, 
spreadsheet) 

Can  the  tool  import  databases? 
Spreadsheets? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Export  (presentation 
graphics) 

Can  the  tool  export  presentation 
graphics,  such  as  charts  and  figures? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Export  (model  to  MS 
Excel) 

Can  the  tool  export  underlying  models  to 
MS  Excel? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

XML 

Can  the  tool  handle  XML  inputs  and/or 
produce  XML  outputs 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

API  (Embedded 
Decision  Support) 

Does  the  tool  have  an  Application 
Programming  Interface  (API;  specifies 
how  software  components  interact  with 
each  other)  so  that  the  tool's 
functionality  can  be  embedded  into 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 
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Definition 


Rationale  for  Inclusion 


Primary  Origin 


other  applications? 


Limited  use  (Run¬ 
time)  version  for 
those  without  the 
software? 


Can  model  structure 
be  copied? 


Can  model  segments 
be  moved  easily 
within  the  model? 


Can  model  structure 
be  displayed  on 
screen? 


Can  model  structure 
be  printed? 


Can  user  protect  data 
from  other  users? 


Can  user  protect 
model  structure  and 
formulae? 


Does  the  software 
support  explicitly 
group  elicitation? 


If  yes,  will  it  support 

decentralized 

elicitation? 


If  yes,  will  it  support 
simultaneous  data 
input? 


If  yes,  does  it  support 

simultaneous 

viewing? 


If  yes,  is  a  record  of 
model  evolution 
kept? 


Capture  descriptive 
comments  or 
narrative  during  TSE. 


Is  a  run-time  version  of  the  tool  available 
so  that  others  without  the  tool  can  run 
models  that  were  generated  by  the  tool? 


Can  the  user  copy  and  paste  parts  of  the 
model  structure  to  limit  the  amount  of 
repetition  in  model  creation? 


Can  the  user  cut  and  paste  parts  of  the 
model  structure  to  limit  the  amount  of 
repetition  in  model  creation? 


Can  the  model  structure  be  displayed 
graphically  for  the  user  to  inspect  and 
understand  ordered  flow  of  information? 


Can  selected  portions  of  the  model  be 
printed? 


Can  the  user  set  permissions  such  that 
certain  data  or  fields  within  the  tool  are 
protected/hidden  from  other  users? 


Can  the  user  set  permissions  such  that 
certain  model  structures  or  formulae 
within  the  tool  are  protected/hidden 
from  other  users? 


Does  the  tool  contain  separate  features 
or  fields  for  group  elicitation  (versus 
single  input)? 


If  the  tool  explicitly  supports  group 
elicitation,  does  it  support  decentralized 
elicitation  where  multiple  users  may  be 
geographically  separated? 


If  the  tool  supports  decentralized 
elicitation  from  multiple,  geographically 
separated  users,  does  it  support 
simultaneous  data  input  from  the  users? 


If  the  tool  supports  simultaneous  data 
input  from  multiple,  distributed  users, 
does  it  support  simultaneous,  real-time 
viewing  of  the  tool  canvas  or  data  entry 
fields? 


If  the  tool  supports  simultaneous, 
distributed  data  input  and  viewing,  does 
it  keep  a  record  of  model  evolution? 


Does  the  tool  allow  capture  of 
written/typed  or  verbal  comments 
during  data  entry  and  subsequent 
viewing  of  the  effect  of  changing  inputs? 


Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

There  is  a  need  to 

document  the  decisions 
that  are  being  made 
throughout  the  TSE  process. 
Analysts  and  decision 
makers  may  "see" 
something  from  a  unique 

ERS  Tradespace 
Workshop 
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Record  TSE  actions 
for  playback  or  re¬ 
creation  of  the 
exploration. 


Automatically 
repopulate  the  TS 
based  on  user- 
selected  seed  points. 


Create  predictive 
models  using 
multiple  techniques 
(e.g.,  regression, 
Kriging,  neural  nets, 
etc.). 


Access  external 
databases  in  real¬ 
time  to  supplement 
internal  data. 


Does  the  tool  allow  the  user  to  record 
TSE  actions  for  the  purposes  of 
rewinding  and  playing  back  (i.e.,  instant 
replay)  as  well  as  rewinding,  playing 
back,  and  entering  into  the  process  to 
take  a  new  TSE  path? 


Can  the  user  "click"  within  or  "point"  to  a 
region  in  the  tradespace  and  set  this 
point  as  a  seed  from  which  the  tool  can 
further  fill  in  or  re-populate  the 
tradespace? 


Can  the  tool  create  surrogate  models  of 
data  such  that  the  data  can  be  replaced 
with  equations? 


Can  the  tool  link,  or  connect,  to  external 
databases  in  order  to  supplement  static 
data  sets  within  the  tool? 


Rationale  for  Inclusion 


perspective  that  is  not 
intuitive  to  others.  They 
may  want  to  capture 
written  or  verbal  comments 
to  explain  the  effect  that 
they  saw  when  changing  an 
input  or  condition. 


TSE  exploration  is 
performed  differently  by 
different  people.  There  are 
multiple  purposes  for 
recording  and  then  playing 
back  the  actions  of  TSE: 
documenting  the  origin  of  a 
decision,  documenting  good 
TSE  practice  for  inclusion  in 
a  "TSE  playbook",  learning 
decision  maker  patterns 
and  intent,  etc.  There  is  also 
a  desire  to  track  and 
visualize  how  the 
tradespace  expands, 
reduces,  or  repopulates 
over  time  as  decisions  are 
made  or  as  new  information 
becomes  available  during 
the  TSE  process. 


The  human  tradespace 
explorer  desires  an  ability  to 
fill  in  or  re-populate  regions 
of  the  tradespace  that  are 
"interesting",  sparse, 
overpopulated,  or 
questionable. 


There  is  a  need  to  generate 
relationships  within  and 
amongst  data  in  the 
tradespace  such  that  users 
can  quickly  re-populate 
regions  of  the  tradespace  or 
explore  new  regions. 
Replacing  data  with  an 
equation,  or  set  of 
equations,  also  enables 
near  real-time  interaction 
with  data  and  dynamic 
analyses. 


TSE  tools  include  the  ability 
to  visualize  datasets,  but 
also  require  the  need  to  link 
to  datasets  that  have 
already  been  populated  by 
others  but  that  may  not 
reside  locally. 


Primary  Origin 


ERS  Tradespace 
Workshop 


ERS  Tradespace 
Workshop 


ERS  Tradespace 
Workshop 
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Attribute 


Definition 


Access  to  cost 
models,  databases, 
or  estimating 
relationships? 


Express  alternate  use 
cases  and 
operational 
scenarios. 


Graphical  and 
quantitative 
assessment  of 
technology 
readiness, 

technology  maturity, 
and  technology 
integration  levels 


Near  real-time 
queries 


Customized  data 
layout 


Can  the  tool  link  to,  or  plug  in  to,  cost 
estimating  resources? 


Can  the  user  generate  alternate  use 
cases  and  operational  scenarios  to 
compare  to  the  baseline? 


Can  the  tool  graphically  and 
quantitatively  express  technology 
readiness,  maturity,  and  integration 
levels? 


Does  the  tool  allow  user  interaction  for 
scenarios,  storylines,  and  other  "what- 
ifs"  that  might  aid  the  decision  maker 
and  analyst  in  exploration  of  impacts  due 
to  constraints,  assumptions,  and 
technology  insertion? 


Can  the  tool  create  a  customized  layout 
of  qualitative  and  quantitative  graphics 
and  data? 


Rationale  for  Inclusion 


As  opposed  to  assessing  the 
lifecycle  cost  of  a  design 
after  it  has  been  chosen 
based  on  performance 
metrics,  ERS  desires  to 
include  this  information  as 
part  of  the  decision  making 
and  TSE  process.  This 
category  starts  with  cost 
but  is  expected  to  expand 
to  include  "ilities"  such  as 
safety,  reliability,  and 
sustainability. 


Alternate  user  scenarios 
and  future  operational 
environments  must  be 
expressed  in  sufficient 
detail  to  support  decision 
making  in  the  TSE  process. 
Capturing  this  information 
will  help  better  understand 
which  scenarios  are  driving 
overall  development  and 
cost  risk.  Also,  these 
"alternate  futures"  will 
enable  assessment  of 
system  resiliency  and 
robustness. 


Programs  are  driven  to 
using  lower  risk  (high 
maturity  and  readiness) 
technologies.  Technology 
development  risk,  lifecycle 
cost,  and  other 
performance  characteristics 
must  be  sufficiently  traded 
in  order  to  make  informed 
decisions  on  product 
development. 


Users  need  the  ability  to 
interact  with  the  tradespace 
by  performing  near  real¬ 
time  queries  based  on 
observations  and  new 
knowledge. 


Decision  makers  want  the 
ability  to  customize  graphic 
layouts  based  on  their 
particular  perspective  of  the 
system  (e.g.,  performance 
tradeoffs,  material 
tradeoffs,  and  lifecycle  cost 
tradeoffs). 


Primary  Origin 


ERS  Tradespace 
Workshop 


ERS  Tradespace 
Workshop 


ERS  Tradespace 
Workshop 


ERS  Tradespace 
Workshop 
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Attribute  | 

1  Definition  | 

1  Rationale  for  Inclusion 

Primary  Origin 

Graphical  decision 
network 

Does  the  tool  allow  for  mapping  or 
linking  decisions  from  the  high-level 
decision  maker  metrics  (e.g.,  cost, 
schedule)  down  to  the  low-level  design 
decisions  (e.g.,  material  selection, 
dimensions/tolerances)? 

Decision  makers  are 
interested  in  knowing  the 
rationale  and 
recommendations  of  the 
design  engineers,  but 
ultimately  view  the 
"goodness"  of  a  design 
through  a  lens  that  contains 
only  a  few  high-level 
metrics.  It  is  anticipated 
that  a  graphical  decision 
network  will  enable  deeper 
understanding  of  decision 
impacts. 

ERS  Tradespace 
Workshop 

Selective  search 

Can  the  user  execute  searches  based  on 
user-specified  selective  dimensions  or 
user-specified  regions  of  the  tradespace? 

More  effective  and  efficient 
search  algorithms  are 
required  in  order  to  support 
the  multiple  dimensions 
envisioned  with  ERS 
tradespaces.  Users  want  the 
ability  to  "coach"  the  search 
based  on  human  knowledge 
and/or  insight. 

ERS  Tradespace 
Workshop 

Generate  Data 
Internally 

Can  the  tool  internally  generate  data 
from  models  (as  opposed  to  requiring 
data  sets  to  be  imported  from  other 
sources  which  execute  models)? 

Executing  a  model  will 
produce  output  data.  This 
ability  can  be  used  for 

Monte  Carlo  simulations 
and  also  for  populating  the 
tradespace  for  deeper 
analysis. 

ARE 

Auto-discover 
relationships  within 
data 

Can  the  tool  automatically  discover, 
identify,  or  reveal  relationships  within  a 
dataset  or  across  multiple  datasets? 

The  human  tradespace 
explorer  may  not  have  the 
ability  to  recognize 
patterns,  groupings,  or 
trends  simultaneously  in 
multiple  dimensions  within 
a  dataset.  There  is  a  need 
for  automatic  recognition  or 
discovery  of  this 
information,  and  then 
displaying  it  to  the  user. 

ARE 

Accept  real-time  user 
input  and  steering 
during  the  TSE 
process. 

Can  the  user  interject  real-time  input  to 
the  TSE  process  (i.e.,  what-ifs  from  other 
collaborators)? 

There  is  a  need  for 
collaborative  TSE.  The  tool 
should  be  able  to  accept 
real-time  changes  or  inputs 
from  any  one  of  the  users 
who  want  to  perform  a 
what-if  analysis,  or  if  a  user 
wants  to  create  a  new 

branch  in  the  TSE  decision 

tree. 

ERS  Tradespace 
Workshop 

Customize  the  TSE 
process,  analysis, 
optimization,  or 
visualization  through 
scripting. 

Does  the  tool  allow  the  user  to  develop 
their  own  custom  code?  For  example, 
multiple  JMP  analyses  can  be  combined 
to  run  in  succession  within  a  single  JMP 
script,  and  VBA  can  be  used  to  create 

There  is  a  need  for  the 
tradespace  explorer  to 
generate  custom  analysis, 
optimization,  and 
visualizations,  beyond  what 

ERS  Tradespace 
Workshop 

67 


:s  of  output  displays  Availability  of  graphical  elicitation  techniques  Decision  Algorithms 

id  analyses  charts  Implemented 


Model  structure  / 
Brainstorming 


Value  functions  / 
scores 


Value  weights 


Probabilities 


Risk  preference 


Swing  weight 
methods 


Can  probabilities  or 
weights  be  defined 
as  variables  that  can 
be  operated  on? 


Can  the  tool 
generate  strategy 
tables? 


Can  the  user 
document  structure 
or  judgments  with 
text? 


Are  graphical 
sensitivity  analyses 
possible  on  either 
weights  or 
probabilities? 


Can  analytical  results 
be  portrayed 
graphically? 


Definition 

Rationale  for  Inclusion 

statistical  analyses  not  available  natively 
in  Excel  or  through  add-ins. 

may  be  available  in  the  out- 
of-the-box  tool,  in  order  to 
make  more  informed 

decisions. 

Does  the  tool  use  any  MODA/MAUT 
algorithms? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  have  built-in  capability  to 
perform  AHP,  including  consistency 
checks? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  use  another  decision 
algorithm  other  than 

MODA/MAUT/AHP?  If  yes,  then  which 
ones? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  use  graphical  techniques 
to  elicit  requirements,  ideas,  thoughts, 
or  model  structure  from  the  user? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  visually  depict  value 
functions  to  elicit  information  from  the 

user? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  visually  depict  value 
weights  to  elicit  information  from  the 
user? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  visually  depict  probabilities 
to  elicit  information  from  the  user? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  visually  depict  risk 
preference  to  elicit  information  from  the 
user? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  implement  swing  weight 
methods  to  graphically  elicit  information 
from  the  user? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  allow  probabilities  or 
weights  to  be  defined  as  variables? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  generate  strategy  tables, 
where  multiple  options  are  listed  for 
multiple  decision  areas? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  allow  users  to  document 
model  structure,  assumptions,  or  other 
judgments  with  text? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  plot  sensitivity  analyses  on 
either  weights  or  probabilities? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

Does  the  tool  plot/chart  analytical 
results? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 


2012  OR/MS 
Today  DAS  Survey 
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Attribute 


Definition 


Rationale  for  Inclusion 


Primary  Origin 


Can  the  tool  produce 
Expected  Value  (EV) 
Tornado  Diagrams? 

Does  the  tool  produce  EV  tornado 
diagrams,  such  that  the  user's  attention 
can  be  called  to  variables  that  have  the 
greatest  impact  on  expected  value? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Does  the  software 
support  two-way 
sensitivity  analysis 

Does  the  tool  determine  how  a 
combination  of  any  two  inputs  affects  an 
output? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Does  the  software 

determine  value  of 

imperfect 

information? 

Does  the  tool  determine  value  of 
imperfect  information  (the  amount  one 
would  be  willing  to  'pay'  for  obtaining 
additional,  uncertain  information)? 

Legacy  OR/MS  Today  DAS 
Survey  Entry 

2012  OR/MS 
Today  DAS  Survey 

Select  data  for 
plotting  and/or 
analysis  through 
filtering,  sorting, 
selecting,  excluding. 

Can  the  user  select  or  exclude  data  from 
an  analysis  or  plot? 

Data  may  be  inaccurate, 
incomplete,  or  not 
appropriate  for  a  specific 
analysis.  The  user  needs  an 
ability  to  perform  these 
actions  on  the  data  when 
conducting  analysis  or 
generating  visualizations. 

Literature  Review 

Visualization  of  data 
through  linked  plots 
(such  as  small 
multiples). 

If  multiple  plots  are  laid  out  on  the 
viewing  canvas,  can  the  tool 
automatically  link  the  plots  such  that 
highlighting  a  point  in  one  plot  results  in 
the  corresponding  points  being 
highlighted  in  all  other  plots? 

There  is  a  need  to  view  data 

in  more  than  2  dimensions 
simultaneously.  One 
method  is  to  view  multiple 

2D  plots  simultaneously  and 
manually  trace  a  point 
(design)  throughout 
multiple  coordinates. 

Literature  Review 
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Intentionally  lelt  blank. 
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Appendix  D.  Mapping  of  TSE  Tools  to  Attributes 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Class 

Usage 

Operating  Systems 

Attribute 

Tool 

Process 

Specific  industries  or 
market  segments  in  which 
the  software  tool  is  more 
popular 

Specific  applications  in 
which  the  software  tool  is 
more  widely  used  for 

User  or  developer 
community  size 

Windows 

Mac  OS 

Unix/Linux 

Flavors  of  Unix/Linux 

Web  implementation 

Definition 

A  piece  of  software, 
proprietary  or  commercial, 
to  conduct  tradespace 
exploration,  analysis, 
and/or  data  visualization. 

An  analysis  method  or 
workflow  for  conducting 
tradespace  exploration, 
analysis,  and/or  data 
visualization. 

Are  there  sectors  of 
industry  that  routinely  use 
the  tool  most  often? 

What  is  the  specific  task  or 
application  that  the  tool  is 
used  for  most  often? 

How  large  and  mature  is 
the  community  who 
develops,  uses,  supports, 
and  advances  this  tool? 

Does  the  tool  operate  in 
Windows  operating 
system? 

Does  the  tool  operate  in 
Macintosh  operating 
system? 

Does  the  tool  operate  in 
Unix  or  Linux  operating 
systems? 

Which  versions  of  Unix  or 
Linux  operating  systems 
can  the  tool  operate  in? 

Can  the  tool  be 
implemented  or  used  from 
the  open  web  (e.g.,  http  or 
https)? 

Tool/Process 

Developer 

1 

10 

AAMODAT 

RDECOM-ARDEC 

Y 

Y 

Military  AoA's 

Decision  analysis  for 
analysis  of  alternatives 
using  VFT 

Small 

Y 

? 

7 

N 

ASEC 

RDECOM-TARDEC 

Y 

Y 

Military 

Management  of  risk  and 
opp.,  decisions, 
capabilities,  and 
requirements 

Small 

Y 

Y 

Y 

Y 

ATSV 

Penn  State  ARL 

Y 

Aerospace,  automotive, 
energy 

Visual  steering, 
histograms,  binned  plots, 
clustering 

Small 

Y 

Y 

7 

N 

CPAT 

PEO-GCS 

Y 

Military 

Establish  optimum  mix  of 
existing  vehicles 

Small 

Y 

? 

7 

N 

Excel 

Microsoft 

Y 

Data  analysis 

Data  visualization, 
plotting,  and  fitting 

Large 

Y 

Y 

Y 

N 

FACT 

GTRI 

Y 

Y 

Military 

Model-based  ground  and 
amphibious  vehicle  design 

Small 

Y 

Y 

Y 

Y 

JMP 

SAS 

Y 

Data  visualization, 
surrogate  modeling, 
statistical  analysis 

Medium 

Y 

Y 

Y 

Suse,  Redhat,  Fedora 

N 

Matlab 

Mathworks 

Y 

Algorithm  development, 
data  analysis,  numeric 
computation 

Large 

Y 

Y 

Y 

N 

ModelCenter 

Phoenix 

Integration 

Y 

Product  Development 
Teams 

Integration  and  analysis  of 
multiple  software  and 
optimization 

Large 

Y 

N 

N 

N 

OpenMDAO 

NASA 

Y 

Engineering 

Multidisciplinary  Design, 
Analysis  and  Optimization 

Small 

Y 

Y 

Y 

Any  with  python 

N 

R 

R  Foundation 

Y 

Financials,  Media 

Statistical  analysis  and 
data  visualizations 

Large 

Y 

Y 

Y 

Debian,  Redhat,  Suse, 
Ubuntu 

N 

TIES 

Georgia  Institute 
of  Technology 

Y 

Academia,  Engineering 

Technology  Identification, 
Sensitivity  Analysis, 
Surrogate  Modeling 

Small 

N/A 

N/A 

N/A 

N/A 

N/A 

WSTAT 

PEO-GCS 

Y 

Y 

Military  AoA's 

Decision  analysis  for 
analysis  of  alternatives 
using  VFT 

Small 

Y 

? 

7 

N 

Pricing  Information 

Training  Classes  Offered 

Attribute 

Other  Operating  Systems 

Commercial 

Education 

Enhanced/High 

Performance 

Open/Free 

Vendor 

3rd  Party 

Classroom 

Online 

Basic 

Definition 

Can  the  tool  operate  in 
operating  systems  not 
listed  above? 

What  is  the  purchase  price 
of  the  standard 

commercial  version  of  the 
tool/process? 

What  is  the  purchase  price 
of  the  education  version 
of  the  tool/ process? 

What  is  the  purchase  price 
of  the  enhanced  or  higher 
performance  (i.e., 

'deluxe')  version  of  the 
tool/process? 

Is  the  tool/process 
considered  Open  Source 
or  Freeware  software; 
available  without  cost  to 
anyone,  public  or 
government? 

Does  the  vendor  offer 
training  classes  for  the 
tool/process? 

Does  a  third  party 
organization  offer  training 
classes  for  the 
tool/process? 

Are  in-classroom  training 
sessions  offered  by  the 
vendor  or  a  third  party? 

Are  online  training 
sessions  offered  by  the 
vendor  or  a  third  party? 

Requires  little  to  no  prior 
knowledge  of  the  software 
interface  or  underlying 
math  to  effectively  use 
and  generate  results. 
Operation  is  intuitive  and 
the  user  can  learn  how  to 

use  it  with  relative  ease. 

Tool/Process 

Developer 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20  1 

AAMODAT 

RDECOM-ARDEC 

N/A 

N/A 

N/A 

Y 

N 

N 

N 

N 

ASEC 

RDECOM-TARDEC 

N/A 

N/A 

N/A 

Y 

Y 

Y 

N/A 

ATSV 

Penn  State  ARL 

N/A 

N/A 

N/A 

Y 

N 

N 

N 

N 

Y 

CPAT 

PEO-GCS 

N/A 

N/A 

N/A 

N/A 

N 

N 

N 

N 

Excel 

Microsoft 

$220 

$140 

$400 

N 

Y 

Y 

Y 

FACT 

GTRI 

N/A 

N/A 

N/A 

Y 

N 

N 

JMP 

SAS 

iOS 

$1,400 

$1,400 

$1,400 

Y 

Y 

Y 

Y 

Matlab 

Mathworks 

iOS,  Android 

$2,150 

$99 

$2,150 

N 

Y 

Y 

Y 

ModelCenter 

Phoenix 

Integration 

"$20,000 

N 

Y 

Y 

Y 

OpenMDAO 

NASA 

N 

N/A 

N/A 

N/A 

Open 

N 

N 

R 

R  Foundation 

N 

N/A 

N/A 

N/A 

Open 

N/A 

Y 

Y 

TIES 

Georgia  Institute 
of  Technology 

N/A 

N/A 

N/A 

N/A 

Y 

N/A 

Y 

N/A 

WSTAT 

PEO-GCS 

N/A 

N/A 

N/A 

Y 

N 

N 

N 

N 

[rraining/Experience  Needed 

Software  Attributes 

Attribute 

Intermediate 

Advanced 

Is  live  or  delayed  response 
help  available  from  the 
developer  or  broader  user 
community 

Are  help  menu,  demos,  or 
tutorials  available 

Limitations  on  the  dataset 
capable  of  being  explored 

Approved  for  use  on  a 
classified  network 

Currently  in  use  by  any  of 
the  ERS  Demonstration 
Project  teams 

Limitations  in  "free" 
software  (e.g.,  toolbox  is 
free  but  host  software  is 
not). 

Standard  or  open  data 
storage,  interchange,  and 

reuse 

Primary  Language  to 

Expand  or  Develop 
Functionality 

Definition 

Requires  prior 
intermediate  level 
knowledge  or  of  the 
subject  matter  to 
effectively  use  the  tool, 
generate  and  interpret 
results.  Minimal  training 
may  be  needed. 

Requires  prior  advanced 
level  knowledge  of  the 
subject  matter  to 
effectively  use  the  tool, 
generate  and  interpret 
results.  Extensive  training 
(possibly  in  the  form  of 
developer  taught  classes) 
may  be  required  to  take 
full  advantage  of  all 
capabilities. 

Can  the  user  reach  out  to 
a  live  person  for  advice  or 
help,  or  is  there  a  user 
forum  that  offers  delayed 
responses  to  questions? 

Is  there  help  available 
within  the  tool,  and  are 
there  demo  videos  or 

tutorials  that  can  aid  in 
learning  how  to  better  and 
more  fully  use  the  tool? 

Are  there  size  (i.e., 
memory,  row,  column, 
etc.)  limitations  to  the 
tool's  capability? 

Is  the  tool  approved  for 
use  on  a  classified 

network? 

Is  the  tool  currently  in  use 
by  any  of  the  ERS 
Demonstration  Project 
teams? 

Are  there  any  limitations 
or  caveats  to  this  tool 
being  "free"? 

Does  the  tool  support 
standard  file  formats  for 
data  storage,  transfer,  and 
reuse? 

What  programming 
language  allows  expansion 
of  the  tool’s  capabilities? 
Include  languages  that 
impact  the  native  tool, 
shared  toolboxes,  and  add¬ 
ins. 

Tool/Process 

Developer 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30  1 

AAMODAT 

RDECOM-ARDEC 

Y 

Y 

N 

Yes;  limited  by  Excel 

N 

N 

Y  (CSV) 

VBA 

ASEC 

RDECOM-TARDEC 

Y 

N 

N 

N 

N 

N 

N 

Y  (SQL) 

HTML 

ATSV 

Penn  State  ARL 

N 

N 

Yes;  limited  by  Excel 

N 

N 

N 

Y  (CSV,  TXT) 

Java 

CPAT 

PEO-GCS 

Y 

N 

N 

Yes;  limited  by  Excel 

N 

N/A 

Y  (CSV) 

VBA 

Excel 

Microsoft 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y  (CSV,  XML) 

VBA 

FACT 

GTRI 

Y 

Y 

Y 

? 

N 

N 

N 

Y(HTML,  XML) 

Java,  Python 

JMP 

SAS 

Y 

Y 

Y 

Y 

Y 

Y(30  day  trial) 

Y  (CSV,  TXT) 

JMP  Scripting  Language 

Matlab 

Mathworks 

Y 

Y 

? 

Y 

N/A 

Y 

Matlab 

ModelCenter 

Phoenix 

Integration 

Y 

Y 

Y 

■> 

Y 

N/A 

Y 

ModelCenter 

OpenMDAO 

NASA 

Y 

Y 

Y 

Y 

Limits  determined  by  local 
machine  capabilities 

N 

N 

N 

Y 

Python 

R 

R  Foundation 

Y 

Y 

Y 

Y 

Limits  determined  by  local 
machine  capabilities 

N 

N 

N 

Y 

R 

TIES 

Georgia  Institute 
of  Technology 

Y 

Y 

N 

Y 

N/A 

N/A 

N 

N/A 

N/A 

N/A 

WSTAT 

PEO-GCS 

Y 

Y 

N 

Yes;  limited  by  Excel 

N 

N 

Y  (CSV) 

Java 

Applications 

Attribute 

Best  option  using  multiple 
objectives 

Representation  /  analysis 
of  uncertainty 

Representation  /  analysis 
of  probabilistic 
dependencies 

Risk  Preference 

Sequential  decision 
making 

Portfolio  decision  making 

Bayesian  Belief  Networks 

Multiple  stakeholders 
collaboration 

Interfaces  to  other 

software: 

Import  (database, 
spreadsheet) 

Definition 

Can  the  tool  select  a  best 
option  from  a  list  of 
existing  options  using 
multiple,  competing 
objectives? 

Can  the  tool  represent 
uncertainty  or  perform 
analysis  of  the  uncertainty 
within  the  model  results? 

Can  the  tool  represent 
probabilistic  dependencies 
or  perform  analysis  of  the 
dependencies  between 
inputs  and  outputs? 

Can  the  tool  model  risk 
preference  of  individual 
stakeholders,  or  of  a 
group? 

Does  the  tool  conduct 
sequential  decision 
making? 

Does  the  tool  conduct 
portfolio  decision  making, 
including  project/process 
portfolio  management  and 
simulation? 

Can  the  tool  create  and/or 
use  Bayesian  belief 
networks? 

Can  the  tool  handle  inputs 
from  multiple 
stakeholders,  either 
through  combining, 
normalizing,  utility 
functions,  or  preferences? 

Can  the  tool  interface  with 
other  software,  either 
manually  or  automatically, 
via  import  or  export  of 
inputs/outputs? 

Can  the  tool  import 
databases?  Spreadsheets? 

Tool/Process 

Developer 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40  1 

AAMODAT 

RDECOM-ARDEC 

Y 

Y 

Y 

N 

N 

Y 

Y 

ASEC 

RDECOM-TARDEC 

N 

Y 

N 

Y 

Y 

N 

Y 

Y 

ATSV 

Penn  State  ARL 

N 

Y 

N 

N 

N 

N 

N 

Y 

Y 

CPAT 

PEO-GCS 

Y 

N 

N 

N 

N 

Y 

Y 

Excel 

Microsoft 

Y 

Y 

Y 

N 

N 

Y 

Y 

FACT 

GTRI 

N 

Y 

Y 

N 

N 

N 

N 

Y 

N 

JMP 

SAS 

N 

Y 

Y 

N 

N 

N 

N 

Y 

Y 

Matlab 

Mathworks 

N 

Y 

N 

N 

N 

N 

Y 

Y 

ModelCenter 

Phoenix 

Integration 

N 

Y 

Y 

N 

N 

N 

N 

Y 

Y 

OpenMDAO 

NASA 

N 

N 

N 

N 

N 

Y 

Y 

Y 

R 

R  Foundation 

N 

Y 

Y 

N 

N 

N 

Y 

N 

Y 

TIES 

Georgia  Institute 
of  Technology 

N 

N 

N 

N 

N 

N 

N 

N 

N 

N 

WSTAT 

PEO-GCS 

Y 

Y 

Y 

Y 

N 

N 

Y 

Y 

Attribute 

Export  (presentation 
graphics) 

Export  (model  to  MS 

Excel) 

XML 

API  (Embedded  Decision 
Support) 

Limited  use  (Run-time) 
version  for  those  without 

the  software? 

Can  model  structure  be 
copied? 

Can  model  segments  be 
moved  easily  within  the 
model? 

Can  model  structure  be 
displayed  on  screen? 

Can  model  structure  be 
printed? 

Can  user  protect  data 
from  other  users? 

Definition 

Can  the  tool  export 
presentation  graphics, 
such  as  charts  and  figures? 

Can  the  tool  export 
underlying  models  to  MS 
Excel? 

Can  the  tool  handle  XML 
inputs  and/or  produce 

XML  outputs 

Does  the  tool  have  an 
Application  Programming 
Interface  (API;  specifies 
how  software  components 
interact  with  each  other) 
so  that  the  tool’s 
functionality  can  be 
embedded  into  other 
applications? 

Is  a  run-time  version  of 

the  tool  available  so  that 

others  without  the  tool 

can  run  models  that  were 
generated  by  the  tool? 

Can  the  user  copy  and 
paste  parts  of  the  model 
structure  to  limit  the 
amount  of  repetition  in 
model  creation? 

Can  the  user  cut  and  paste 
parts  of  the  model 
structure  to  limit  the 
amount  of  repetition  in 
model  creation? 

Can  the  model  structure 
be  displayed  graphically 
for  the  user  to  inspect  and 
understand  ordered  flow 

of  information? 

Can  selected  portions  of 
the  model  be  printed? 

Can  the  user  set 
permissions  such  that 
certain  data  or  fields 

within  the  tool  are 
protected/hidden  from 
other  users? 

Tool/Process 

Developer 
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N 

N 
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N 

Software  Features 

Attribute 

Can  user  protect  model 
structure  and  formulae? 

Does  the  software  support 
explicitly  group  elicitation? 

If  yes,  will  it  support 
decentralized  elicitation? 

If  yes,  will  it  support 
simultaneous  data  input? 

If  yes,  does  it  support 
simultaneous  viewing? 

If  yes,  is  a  record  of  model 
evolution  kept? 

Capture  descriptive 
comments  or  narrative 
during  TSE. 

Record  TSE  actions  for 
playback  or  re-creation  of 
the  exploration. 

Automatically  repopulate 
the  TS  based  on  user- 
selected  seed  points. 

Create  predictive  models 
using  multiple  techniques 
(e.g.,  regression,  Kriging, 
neural  nets,  etc.). 

Definition 

Can  the  user  set 
permissions  such  that 
certain  model  structures 

or  formulae  within  the 
tool  are  protected/hidden 
from  other  users? 

Does  the  tool  contain 
separate  features  or  fields 
for  group  elicitation 
(versus  single  input)? 

If  the  tool  explicitly 
supports  group  elicitation, 
does  it  support 
decentralized  elicitation 
where  multiple  users  may 
be  geographically 
separated? 

If  the  tool  supports 
decentralized  elicitation 
from  multiple, 
geographically  separated 
users,  does  it  support 
simultaneous  data  input 
from  the  users? 

If  the  tool  supports 
simultaneous  data  input 
from  multiple,  distributed 
users,  does  it  support 
simultaneous,  real-time 
viewing  of  the  tool  canvas 
or  data  entry  fields? 

If  the  tool  supports 
simultaneous,  distributed 
data  input  and  viewing, 
does  it  keep  record  of 
model  evolution? 

Does  the  tool  allow 
capture  of  written/typed 
or  verbal  comments 
during  data  entry  and 
subsequent  viewing  of  the 
effect  of  changing  inputs? 

Does  the  tool  allow  the 

user  to  record  TSE  actions 
for  the  purposes  of 
rewinding  and  playing 
back  (i.e.,  instant  replay) 
as  well  as  rewinding, 
playing  back,  and  entering 
into  the  process  to  take  a 
new  TSE  path? 

Can  the  user  "click"  within 
or  "point"  to  a  region  in 
the  tradespace  and  set 
this  point  as  seed  from 
which  the  tool  can  further 
fill  in  or  re-populate  the 
tradespace? 

Can  the  tool  create 
surrogate  models  of  data 
such  that  the  data  can  be 
replaced  with  equations? 
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Attribute 

Access  external  databases 
in  real-time  to  supplement 
internal  data. 

Access  to  cost  models, 
databases,  or  estimating 
relationships? 

Express  alternate  use 
cases  and  operational 
scenarios. 

Graphical  and  quantitative 
assessment  of  technology 
readiness,  technology 
maturity,  and  technology 
integration  levels 

Near  real-time  queries 

Customized  data  layout 

Graphical  decision 
network 

Selective  search 

Generate  Data  Internally 

Auto-discover 
relationships  within  data 

Definition 

Can  the  tool  link,  or 
connect,  to  external 
databases  in  order  to 
supplement  static  data 
sets  within  the  tool? 

Can  the  tool  link  to,  or 
plug  in  to,  cost  estimating 
resources? 

Can  the  user  generate 
alternate  use  cases  and 
operational  scenarios  to 
compare  to  the  baseline? 

Can  the  tool  graphically 
and  quantitatively  express 
technology  readiness, 
maturity,  and  integration 
levels? 

Does  the  tool  allow  user 
interaction  for  scenarios, 
storylines,  and  other 
"what-ifs"  that  might  aid 
the  decision  maker  and 
analyst  in  exploration  of 
impacts  due  to 
constraints,  assumptions, 
and  technology  insertion? 

Can  the  tool  create  a 
customized  layout  of 
qualitative  and 
quantitative  graphics  and 
data? 

Does  the  tool  allow  for 
mapping  or  linking 
decisions  from  the  high- 
level  decision  maker 
metrics  (e.g.,  cost, 
schedule)  down  to  the  low- 
level  design  decisions 
(e.g.,  material  selection, 
dimensions/tolerances)? 

Can  the  user  execute 

searches  based  on  user- 
specified  selective 
dimensions  or  user- 
specified  regions  of  the 
tradespace? 

Can  the  tool  internally 
generate  data  from 
models  (as  opposed  to 
requiring  data  sets  to  be 
imported  from  other 
sources  which  execute 
models)? 

Can  the  tool  automatically 
discover,  identify,  or 
reveal  relationships  within 
a  dataset  or  across 
multiple  datasets? 
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Decision  Algorithms  Implemented 

Availability  of  graphical  elicitation  te 

Attribute 

Accept  real-time  user 
input  and  steering  during 
the  TSE  process. 

Customize  the  TSE 
process,  analysis, 
optimization,  or 
visualization  through 
scripting. 

MODA/MAUT 

AHP 

Other 

Model  structure  / 
Brainstorming 

Value  functions  /  scores 

Value  weights 

Probabilities 

Risk  preference 

Definition 

Can  the  user  interject  real¬ 
time  input  to  the  TSE 
process  (i.e.,  what-ifs  from 
other  collaborators)? 

Does  the  tool  allow  the 
user  to  develop  their  own 
custom  code?  For 
example,  multiple  JMP 
analyses  can  be  combined 
to  run  in  succession  within 
a  single  JMP  script,  and 

VBA  can  be  used  to  create 
statistical  analyses  not 
available  natively  in  Excel 
or  through  add-ins. 

Does  the  tool  use  any 
MODA/MAUT  algorithms? 

Does  the  tool  have  built-in 
capability  to  perform  AHP, 
including  consistency 
checks? 

Does  the  tool  use  another 
decision  algorithm  other 
than  MODA/MAUT/AHP? 

If  yes,  then  which  ones? 

Does  the  tool  use 
graphical  techniques  to 
elicit  requirements,  ideas, 
thoughts,  or  model 
structure  from  the  user? 

Does  the  tool  visually 
depict  value  functions  to 
elicit  information  from  the 

user? 

Does  the  tool  visually 
depict  value  weights  to 
elicit  information  from  the 

user? 

Does  the  tool  visually 
depict  probabilities  to 
elicit  information  from  the 

user? 

Does  the  tool  visually 
depict  risk  preference  to 
elicit  information  from  the 

user? 
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:hniques 

Types  of  output  displays  and  analyses  charts 

Attribute 

Swing  weight  methods 

Can  probabilities  or 
weights  be  defined  as 
variables  that  can  be 
operated  on? 

Can  the  tool  generate 
strategy  tables? 

Can  the  user  document 
structure  or  judgments 
with  text? 

Are  graphical  sensitivity 
analyses  possible  on  either 
weights  or  probabilities? 

Can  analytical  results  be 
portrayed  graphically? 

Can  the  tool  produce 
Expected  Value  (EV) 
Tornado  Diagrams? 

Does  the  software  support 
two-way  sensitivity 
analysis 

Does  the  software 

determine  value  of 
imperfect  information? 

Select  data  for  plotting 
and/or  analysis  through 
filtering,  sorting,  selecting, 
excluding. 

Definition 

Does  the  tool  implement 
swing  weight  methods  to 
graphically  elicit 
information  from  the 

user? 

Does  the  tool  allow 
probabilities  or  weights  to 
be  defined  as  variables? 

Does  the  tool  generate 
strategy  tables,  where 
multiple  options  are  listed 
for  multiple  decision 
areas? 

Does  the  tool  allow  users 

to  document  model 
structure,  assumptions,  or 
other  judgments  with 
text? 

Does  the  tool  plot 
sensitivity  analyses  on 
either  weights  or 
probabilities? 

Does  the  tool  plot/chart 
analytical  results? 

Does  the  tool  produce  EV 
tornado  diagrams,  such 
that  the  user’s  attention 

can  be  called  to  variables 
that  have  the  greatest 
impact  on  expected  value? 

Does  the  tool  determine 
how  a  combination  of  any 
two  inputs  affects  an 
output? 

Does  the  tool  determine 
value  of  imperfect 
information  (the  amount 
one  would  be  willing  to 
'pay'  for  obtaining 
additional,  uncertain 
information}? 

Can  the  user  select  or 

exclude  data  from  an 
analysis  or  plot? 
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Attribute 

Visualization  of  data 
through  linked  plots  (such 
as  small  multiples). 

Definition 

If  multiple  plots  are  laid 
out  on  the  viewing  canvas, 
can  the  tool  automatically 
link  the  plots  such  that 
highlighting  a  point  in  one 
plot  results  in  the 
corresponding  points 
being  highlighted  in  all 
other  plots? 
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Appendix  E.  ERS  Demonstration  Projects 
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Engineered  Resilient  Systems  (ERS)  Demonstration  Project  1:  Fixed  Wing  Air¬ 
craft  seeks  to  demonstrate  "the  use  of  high  performanee  eomputing  and  physies- 
based  modeling  to  rapidly  develop  a  resilient  design  spaee  during  early  Analysis  of 
Alternatives  of  a  fixed  wing  air  vehiele."^  The  approaeh  is  to  link  design  variables 
with  mission  eapabilities  through  physies-based  and  operational  models,  respee- 
tively.  Aireraft  design  inputs  are  mapped  to  operational  and  eost  analysis  modeling 
and  simulation  (M&S)  outputs.  The  notional  steps  and  eorresponding  tools  used 
during  the  projeet  are  as  follows  and  as  illustrated  in  Fig.  E-1. 


1 .  Consider  multiple  mission  seenarios  and  keep  their  requirements  open  (Tool: 
not  applieable  (N/A) — Joint  Capabilities  Integration  and  Development  Sys¬ 
tem  Doeuments). 

2.  Exeeute  operational  effeetiveness  models  against  multiple  seenarios  and  threats 
(Tool:  Analysis  of  Mobility  Platforms  [AMP]). 

3.  Identify  set  of  operational  performanee  metries  and  system  design  variables 
that  impaet  operational  requirements  (Tools:  DaVinei  and  AMP). 

4.  Conduet  multidiseipline  physies-based  analysis  of  designs.  (Tools:  DaVinei 
and  Kestrel). 

5.  Create  surrogate  response  surfaee  models  that  aeeurately  represent  the  mul- 
tiphysies  based  modeling  design  tool  outputs  and  injeet  it  into  engagement 
models  to  show  an  iterative  ability  to  adjust  seenarios  and  requirements  to 
physieal  feasibility  (Tool:  IMP). 

6.  Perform  a  struetured  assessment  of  eost,  sehedule,  and  performanee  risk  us¬ 
ing  probability  based  design  methods  to  statistieally  eonneet  operational  re¬ 
quirements  and  eoneept  feasibility  with  performanee  and  affordability  (Tool: 
IMP). 

'Eslinger  OJ.  Draft  engineered  resilient  systems  program  management  plan.  Vicksburg  (MS): 

Army  Engineer  Research  and  Development  Center  (US);  2013. 
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Fig.  E-1  TSE  steps  for  DPI 


ERS  DP2:  Ship  Design  seeks  to  address  the  historieal  point-based  design  method 
used  by  the  US  Navy  through  eomparison  to  set-based  praetiee  employing  physios 
based  modeling  and  tradespace  exploration.  In  the  status  quo  approach,  key  design 
parameters  are  locked  down  early  to  enable  continuation  into  detailed  design  and 
manufacturing.  However,  the  design  parameter  decisions  made  early  in  the  design 
process  are  based  on  low  fidelity  information,  when  the  design  is  immature  and 
uncertainty  is  high.^  As  the  point  design  process  progresses,  details  emerge  about 
the  design  and  physics-based  analysis  is  conducted,  resulting  in  identification  of 
design  deficiencies.  To  counter  these  design  deficiencies,  redesign  commences  or 
requirements  are  relaxed.  Continuing,  the  point  design  process  struggles  to  keep 
the  ship  design  feasible  given  the  early  decisions  that  have  already  locked  in  key 
dimensions,  and  the  design  is  unable  to  respond  to  changes.  The  proposed  approach 
is  centered  around  set-based  design,  where  design  variables  are  parameterized  and 
decisions  are  postponed  until  sufficient  detail  is  available.  As  the  sets  of  designs 
narrow  down,  the  best  solution  emerges.  Set-based  design  applied  here  combines 
physics-based  data  with  analysis  of  requirements,  concept  of  operations,  measures 
of  effectiveness,  technology  portfolios,  and  cost  analysis.  The  notional  steps  and 
corresponding  tools  used  during  the  project  are  as  follows  and  as  illustrated  in  Fig. 
E-2. 
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1.  Obtain  a  baseline  ship  design  (Tool:  Advaneed  Ship  and  Submarine  Evalua¬ 
tion  Tool  [ASSET],  N/A — expert  input). 

2.  Identify  set  of  performanee  and  operational  metries  (Tool:  ASSET). 

3.  Synthesize  design  alternatives  and  eolleet  output  data  (Tools:  Rapid  Ship  De¬ 
sign  Environment  [RSDE],  ASSET,  Eeading  Edge  Arehiteeture  for  Prototyp¬ 
ing  Systems  [EEAPS]). 

4.  Analyze  and  optimize  alternative  designs  aeross  performanee,  eost,  and  risk 
(Tools:  RSDE,  ASSET,  LEAPS,  PBCM,  Exeel,  TIGER,  SEAQUEST,  Mod- 
elCenter). 

5.  Identify  the  redueed  set  of  eompliant  designs  based  on  the  bounds  applied  to 
design  variables  and  metries  (Tools:  RSDE,  Exeel). 


Fig.  E-2  TSE  steps  for  DP2 


DP3:  Sensors  Systems  DPS  is  based  on  the  Geo-Environmental  Taetieal  Simula¬ 
tion  program,  whieh  is  an  Army  Teehnology  Objeetive  exeeuted  by  the  US  Army 
Engineer  Researeh  and  Development  Center.  The  objeetive  is  to  apply  "HPC  mod¬ 
eling  &  simulation  to  improve  the  deteetion  of  buried  targets  aeross  a  number  of 
sensing  modes."'  Targets  inelude  improvised  explosive  deviees,  explosively  formed 
penetrators,  land  mines,  weapon  systems,  and  eaehes  of  various  sorts. 
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The  M&S  efforts  on  DPS  seek  to  understand  the  full  impact  of  disparate  geo- 
environmental  factors  (i.e.,  solar  conditions,  weather,  soil  characteristics,  vegeta¬ 
tion,  and  natural/man-made  clutter)  on  the  performance  of  electro-optical/infrared 
and  ground-penetrating  radar  sensor  systems. 

DPS  will  employ  virtual  reality  tools  to  simulate  landscapes,  scenes,  sensors,  and 
endogenous  factors,  to  infer  correlations.  The  virtual  environment  will  allow  multi¬ 
ple  users  to  interact  within  the  scene,  choose  how  to  view  the  data  (i.e.,  via  walking, 
ground  vehicles,  or  air  vehicles),  and  understand  how  different  sensor  platforms 
would  perform  under  various  conditions. 
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Appendix  F.  Capability  Portfolio  Analysis  Tool 
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The  Capability  Portfolio  Analysis  Tool  (CPAT)  was  developed  to  provide  a  formal, 
structured  process  to  address  the  difficulty  associated  with  the  decision  analysis 
process  within  fleet  modernization.  CPAT  follows  a  value-focused  thinking  (VFT) 
approach,  which  involves  decomposing  weighted  roles  into  functions  and,  using  a 
mixed  integer  linear  programming  optimization  model,  develops  portfolios  of  sys¬ 
tems  that  maximize  fleet  value  over  time  while  meeting  cost  and  schedule  con¬ 
straints.  The  following  general  steps  are  conducted  within  CPAT  to  conduct  tradespace 
exploration  (TSE).  An  overview  description  of  CPAT  is  available  in  Appendix  A, 
in  the  CPAT  section.  Outside  resources  on  CPAT  are  also  available. 

The  process  within  CPAT  generally  follows  the  following  steps: 

1 .  Gather  information 

•  Alternative  and  baseline  specifications  (system,  subsystem,  and  compo¬ 
nent  level) 

•  Mission  scenarios  and  requirements  (operational  and  system  level) 

•  Measures  of  effectiveness,  measures  of  performance,  key  performance 
parameters,  key  system  attributes,  and  "ilities" 

•  Data  and  assumptions  (schedule,  base  year,  budget  available,  period  of 
operation) 

2.  Use  expert  knowledge  (subject  matter  experts,  project  managers,  and  expert 
knowledge  teams)  to  conduct  VFT 

•  Break  down  the  mission  scenarios  and  requirements  into  functional  roles 

•  Assign  values  to  each  role 

•  Determine  major  system  attributes  that  contribute  to  each  value 

•  Map  measures  to  each  system  attribute,  and  weight  their  importance 
using  a  swing  weight  matrix 

'Edwards  S,  Haas  B.  Program  executive  office  ground  combat  systems  capability  portfolio 
analysis  tool  (CPAT).  In:  Proceedings  of  the  79th  MORS  Symposium,  WG  27  -  Decision  Analysis; 
201 1  Jun  20-23;  Monterey,  CA.  MORSS;  2006. 

^Ewing  L,  Dell  RE,  MacCalman  M,  Whitney  L.  Capability  portfolio  analysis  tool  (CPAT)  ver¬ 
ification  and  validation  report.  Monterey  (CA):  Naval  Postgraduate  School;  2013  Jan.  Report  No.: 
NPS-OR-13-001. 

^Edwards  S.  Capability  portfolio  analysis  tool  (CPAT)  &  whole  system  trade  analysis  (WSTA). 
Paper  presented  at:  Army  Systems  Engineering  Eomm  (ASEE).  2013  Eeb  27;  Arlington,  VA. 
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•  Determine  value  functions;  value  functions  may  be  dependent  on  the 
system  performance  data  (alternatives  and  baselines  specifications) 

3.  Perform  life  cycle  cost  (LCC)  estimates  of  each  alternative  based  on  possible 
courses  of  action  (COAs) 

•  Use  data  and  assumptions  data  when  applicable 

•  For  each  alternative,  perform  a  year-by-year  LCC  estimate 

•  Compare  the  resulting  LCC  estimates  for  the  different  alternatives 

•  Based  on  the  comparisons,  data,  and  assumptions,  determine  feasible 
COAs 

4.  Perform  a  schedule  and  life  cycle  analysis 

•  Gather  research,  development,  test,  and  evaluation  (RDT&E)  and  acqui¬ 
sition  budget  schedule  and  distribution 

•  Determine  expected  availability  date  of  modernization  alternatives 

•  Determine  the  required  date  for  required  funds  to  procure  systems 

•  Determine  dates  the  alternatives  start  realizing  performance  improve¬ 
ments 

•  Map  the  above  to  life  cycle  phases  (RDT&E,  procurement,  operation, 
and  retirement) 

5.  Perform  an  optimization  analysis  to  determine  the  optimum  alternatives  for 
each  COA 

•  Gather  all  requirements  and  constraints 

•  Determine  the  number  of  systems  for  each  role  (if  portfolios  are  being 
analyzed) 

•  Peed  all  performance,  cost,  and  schedule  data  into  the  CPAT  optimiza¬ 
tion  routine  and  map  to  objectives  and  constraints  for  each  COA 

•  Pormulate  the  problem  as  an  optimization  problem  and  select  the  appro¬ 
priate  solver  to  be  used  to  find  the  optimum  solution 

•  Collect  optimization  results  (year-by-year  schedule  of  systems  or  port¬ 
folio  alternatives) 

6.  Perform  postoptimization  analyses 
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•  Assess  the  operational  impact  of  the  alternatives  by  looking  at  how  each 
alternative  in  each  COA  meets  the  roles,  and  compare  the  different  op¬ 
tions 

•  Perform  an  industrial  base  assessment  by  gathering  industrial  base  data, 
investment  recommendations  per  role,  and  minimum  sustaining  rate, 
and  determine  if  the  alternatives  meet  the  industrial  constraints 

•  Perform  a  gap  analysis  by  comparing  the  performance  gaps  of  the  cur¬ 
rent  baseline  systems  or  portfolio  with  the  systems  or  portfolio  capa¬ 
bilities  of  the  optimum  systems,  considering  the  performance,  cost,  and 
schedule 

Figure  F-1  is  a  ARL-generated  schematic  of  the  CPAT  TSE  environment,  including 
steps,  inputs,  outputs,  interfaces,  and  analysis  details. 
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Appendix  G.  Framework  for  Assessing  Cost  and  Technology 
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The  Framework  for  Assessing  Cost  and  Teehnology  (FACT)  was  developed  to  pro¬ 
vide  a  formal,  struetured  proeess  for  presenting  engineering  and  aequisition  deei- 
sion  makers  with  eapability,  eost,  and  performanee  trades  on  systems,  using  the  sys¬ 
tem’s  physieal  arehiteeture  and  quantitative  system  attributes.  FACT  was  developed 
as  a  eollaborative  environment  for  eondueting  model-based  systems  engineering 
over  the  Internet  for  the  purpose  of  inereasing  early  knowledge  of  system  capabil¬ 
ity  and  tradeoffs.  An  overview  description  of  FACT  is  available  in  Appendix  A,  in 
the  FACT  section.  Outside  resources  on  FACT  are  also  available. 

The  process  within  FACT  generally  follows  these  steps: 

1.  Develop  or  obtain  system  requirements  and  translate  into  systems  modeling 
language  (SysML)  requirement  models 

•  Requirements  are  preserved  and  updated,  while  maintaining  traceability 
to  verification 

2.  Define  or  obtain  the  system  work  breakdown  structure  (WBS)  and  translate 
into  a  SysML  block  definition  diagram 

•  The  relationships  and  interfaces  within  the  system  hierarchy  are  main¬ 
tained  and  traced 

3.  Identify  alternative  subsystems 

•  Identify  the  attributes  that  define  these  subsystems  and  enter  quantitative 
values 

•  Map  these  subsystem  attributes  to  the  system  requirements 

•  Assign  qualitative  weighting  to  a  subsystem  attribute’s  influence  on  a 
system  requirement 

4.  Develop  a  SysML  parametric  diagram  using  parametric  blocks 

•  Inputs,  outputs,  constraints,  units,  and  calculations  of  the  parametric 
blocks  are  captured  in  SysML  for  linking  to  other  blocks 

'Ender  TR,  Browne  CD,  Yates  WW,  O’Neal  M.  FACT:  an  M&S  framework  for  systems  engi¬ 
neering.  Paper  presented  at:  Interservice/Industry  Training,  Simulation,  and  Education  Conference 
(I/ITSEC).  2012  Dec  3-6;  Orlando,  EL. 

^Ender  TR.  Model-based  systems  engineering  as  a  collaborative  web-service.  Paper  presented 
at:  National  Defense  Industrial  Association  Ground  Vehicle  Systems  Engineering  and  Technology 
Symposium  (NDIA  GVSETS).  2013  Aug  20-22;  Troy,  MI. 
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•  FACT  can  perform  internal  calculations  using  surrogate  models,  or  can 
call  out  to  externally  stored  and  executed  simulations  or  surrogate  mod¬ 
els 

5.  Conduct  analysis 

•  Configure  the  system  by  selecting  subsystems  to  satisfy  the  necessary 
WBS  level  and  observe  impact  on  performance  requirements 

•  Data  are  exported  to  the  collaborative  web  server 

6.  Access  graphical  user  interface  and  conduct  exploration 

•  Adjust  subsystem  performance  attributes  as  necessary  to  reflect  "what- 
if’s" 

•  Apply  uncertainty  to  subsystem  attributes  and  run  Monte  Carlo  simula¬ 
tion 

•  Multiple  users  can  simultaneously  access  and  view  the  tradespace  en¬ 
vironment,  making  changes  to  threshold  and  objectives  in  real-time  as 
well  as  filtering  data 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


AAMODAT 

Armament  Analysis  Multiple  Objeetives  Deeision  Analysis  Tool 

AMP 

Analysis  of  Mobility  Platforms 

ARDEC 

US  Army  Armament  Researeh,  Development  and  Engineering 

Center 

ARE 

US  Army  Researeh  Eaboratory 

ASDE 

Aerospaee  Systems  Design  Eaboratory 

ASEC 

Advaneed  Systems  Engineering  Capability 

ASSET 

Advaneed  Ship  and  Submarine  Evaluation  Tool 

ATSV 

ARE  Trade  Spaee  Visualizer 

CAC 

eommon  aeeess  eard 

CART 

Classifieation  and  Regression  Trees 

CDE 

eumulative  distribution  funetion 

CED 

Computational  Eluid  Dynamies 

COA 

eourse  of  aetion 

CPAT 

Capability  Portfolio  Analysis  Tool 

DAKOTA 

Design  Analysis  toolKit  for  Optimization  and  Teraseale 
Applieations 

DAS 

Deeision  Analysis  Software 

DOD 

Department  of  Defense 

DOE 

design  of  experiments 

DP 

demonstration  projeet 

ERDC 

US  Army  Engineer  Researeh  and  Development  Center 

ERS 

Engineered  Resilient  Systems 
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FACT 

Eramework  for  Assessing  Cost  and  Technology 

FEA 

finite  element  analysis 

FS 

functional  skeleton 

GRIPS 

Genetic  Resources  for  Innovation  and  Problem  Solving 

GTRI 

Georgia  Tech  Research  Institute 

GUI 

graphical  user  interface 

JIAT 

Joint  Integrated  Analysis  Tool 

LCC 

life  cycle  cost 

LEAPS 

Reading  Edge  Architecture  for  Prototyping  Systems 

LMI 

Eogistics  Management  Institute 

M&S 

modeling  and  simulation 

MARS 

Multivariate  Adaptive  Regression  Splines 

MBSE 

model-based  systems  engineering 

MDAO 

Multidisciplinary  Design  Analysis  and  Optimization 

MDO 

multidisciplinary  optimization 

MIT 

Massachusetts  Institute  of  Technology 

NASA 

National  Aeronautics  and  Space  Administration 

N/A 

not  applicable 

O&S 

operations  and  support 

OR/MS 

Operations  Research/Management  Sciences 

PDE 

probability  density  function 

PEO-GCS 

Program  Executive  Office  Ground  Combat  Systems 

PSC 

Priority  Steering  Council 

PSU-ARE 

Pennsylvania  State  University  Applied  Research  Eaboratory 
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QFD 

quality  function  deployment 

RDT&E 

research,  development,  test,  and  evaluation 

RICH 

Rank  Inclusion  in  Criteria  Hierarchies 

ROSETTA 

Relational- Oriented  Systems  Engineering  and  Technology 
Tradeoff  Analysis 

RPM 

Robust  Portfolio  Management 

RSDE 

Rapid  Ship  Design  Environment 

RSE 

response  surface  equation 

SAE 

Systems  Analysis  and  Experimentation 

SAE 

Systems  Analysis  Eaboratory 

SCAP 

System  Capabilities  Analytic  Process 

SEAD 

ARE  Survivability/Eethality  Analysis  Directorate 

SPIDR 

Systems  Platform  for  Integrated  Design  in  Realtime 

SUE 

single  utility  function 

SysME 

systems  modeling  language 

TARDEC 

US  Army  Tank  Automotive  Research,  Development  and 
Engineering  Center 

TIES 

Technology  Identification,  Evaluation,  and  Selection 

TRACER 

Tradespace  Analysis  for  Capabilities,  Effectiveness,  and 

Resources 

TSE 

tradespace  exploration 

VBA 

Visual  Basic  for  Applications 

VET 

value-focused  thinking 

VisualDOC 

Visual  Design  &  Optimization  Control 

VTD 

Vehicle  Technology  Directorate 
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WBS 

WSTAT 


work  breakdown  structure 
Whole  System  Trades  Analysis  Tool 
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